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CHAPTER I 


INTRODUCTION 

1.1 The Marshall Center and the Future Rocket 

The future of space travel is evolving as NASA designs the nation’s next 
generation launch system that will provide the capability for human exploration 
missions beyond low-Earth orbit (i.e., to the moon, Mars and beyond). NASA’s 
Marshall Space Flight Center (referred to hereafter as the Marshall Center) is 
leading the design, development and delivery of the most powerful rocket ever 
built - the Space Launch System or SLS. This advanced, heavy-lift vehicle will 
launch humans and robotic explorers deeper into the solar system than ever 
before. 

This is not unfamiliar territory for NASA and the Marshall Center. The 
Marshall Center is an experienced developer and integrator of launch systems 
possessing the engineering capabilities to take hardware from concept to 
preliminary design to operation in space. Prior to the SLS Program, the Marshall 
Center was responsible for the design and development of the Ares I and Ares V 
launch vehicles within another national-level program called Constellation. The 
Marshall Center was an active participant in the first, successful dedicated vehicle 
and ground test flight of the Ares I-X rocket prior to the cancellation of the 
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Constellation Program. Similarities between the Constellation Program and SLS 
Program are: 

> Share a primary goal of enabling human exploration beyond Low Earth Orbit 
(LEO) 

> Share a common service intent with rockets under design to be the Space 
Shuttle’s successor 

> Share a distributed team approach across multiple NASA Centers 

> Share a common charge to reconstitute systems engineering capacity within 
NASA’s human spaceflight community to smoothly transition the human 
spaceflight workforce to the next generation of capabilities and to lay the 
foundation of a program that will be cost-effective and sustainable into the far 
future (Rhatigan, et ah, 201 1). 

While large scale, distributed programs like Constellation afforded 
advantages such as accessibility to the entire Agency’s technical depth, skills and 
expertise, best practices and approaches, and state-of-the-art NASA facilities and 
infrastructures across the nation, there were also some pivotal disadvantages 
(Constellation Lessons Learned, 2011). There was persistent political tension 
between what was most efficient for the Constellation Program versus what was 
best for a particular NASA Center to sustain or grow its current role. The large 
dispersed teams at ten NASA Centers led to unclear roles and responsibilities and 
exposed cultural differences. With each Center documenting Constellation 
requirements, procedures and processes - oftentimes duplicative and 
contradictory - the contractors had difficulty distinguishing final decisions 
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coming from the Agency, since direction could come from the program or the 
independent technical authority. 

1.2 Lessons from Constellation 

In 201 1, NASA published an executive summary containing lessons 
learned collected from the aerospace workforce who contributed to the 
Constellation Program. From the key findings, the most difficult and most 
persistent challenges involved cost, schedule, and organization (Constellation 
Lessons Learned, 2011). While the NASA Agency is renowned for technical 
prowess, senior managers in flagship programs can be faced with multitude of 
non-technical challenges for which they have far less training or preparation. In 
this respect, using Constellation as a comparative and its lessons learned can 
provide invaluable sources of insight (Constellation Lessons Learned, 2011). For 
the purposes of this study, three major lessons learned pertaining to roles and 
responsibilities, decision-making, and communication are listed and analyzed as 
the basis for the research objectives within this study. 

The clarity of RR&A for the Constellation Program was degraded by the 
combined effects of the wide distribution of program responsibilities via the “10 
Healthy Centers” policy, the multi-decadal phasing of the program development, 
and the assumption of traditionally understood roles from the Space Shuttle 
heritage component development. There is no formula or checklist for clear 
RR&A in an Agency-wide flagship program, but RR&A can be improved by 
periodic functional examination, by either combining like tasks or separating 
functions by needs (Constellation Lessons Learned, 2011). 
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The clarity and effectiveness of the decision-making processes for the 
Constellation Program were driven by the same events as for the RR&A 
mentioned above. In spite of constant attention from senior management, the 
decision-making process remained a persistent issue that only marginally 
improved over time. In a program of this magnitude, attempts to balance timely 
decision-making at the appropriate levels, consider tactical viewpoints and clearly 
delineate accountability for execution, while keeping all stakeholders informed 
and included, often left someone dissatisfied. For any large scope, distributed 
program like Constellation, it is recommended to invest the time and energy to 
define a comprehensive strategic decision process that includes all affected 
parties. Project Planning is vitally important (Constellation Lessons Learned, 
2011 ). 

As mentioned previously, Constellation’s widespread 10-Center team 
created a true communications challenge. While countless assessments and 
prevailing programmatic wisdom indicate a small, centrally located team is the 
most efficient way to build a complex element, Constellation did not have that 
luxury. Thus, this posed RR&A, decision-making and communication issues. 

The Constellation Program incorporated Information Technology (IT) tools and 
applications (telecom, WebEx, Integrated Collaborative Environment (ICE) 
portal, etc.) extensively to enhance the flow of information (Constellation Lessons 
Learned, 2011). 

The vast aerospace industry as well as scientists and academics across the 
globe are watching NASA, tracking its progress in the development of the SLS 
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heavy-lift vehicle. Congress and the U.S. taxpayers are also closely observing its 
progress and alignment with budget and schedule constraints. On the heels of the 
abruptly cancelled Constellation Program in 2010, with not one, but two test 
flights accomplished for the Ares I-X rocket and Orion crew vehicle, 
NASA/Marshall Center must learn vital lessons from Constellation (and other 
large-scale human spaceflight programs such as Apollo, Space Shuttle, 
Shuttle/Mir and International Space Station (ISS)) and adequately apply them to 
the management and development of the SLS Program. 

1.3 Research Questions 

Based on the nature of the responsibility and the watchful eye of the 
nation on NASA in the development of the SLS launch vehicle, the following 
questions will be researched and analyzed in this study: 

> Is the SLS Program applying these lessons learned from the Constellation 
Program to its decision making and communication processes? 

> Does the SLS Program balance timely decision making at appropriate levels? 

> Does the SLS Program make strategic decisions and have a comprehensive 
decision process? 

> How effective is the SLS Program at making decisions? 

> Does a decision making process really matter? 

> How are decisions made on the SLS Program? 

> Are the decisions made by group consensus or directed by management? 

> How is the communication flow within the program? 

> Are all parties included in the decision making and communication process? 
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Consequently the resulting research objectives from the above questions are: 

♦♦♦ Define strategic decisions 

♦♦♦ Define the dimensions of strategic decision making 
♦♦♦ Investigate how decisions are made 

♦♦♦ Investigate the effectiveness of the strategic decision making process 

♦♦♦ Determine if decision making process matters 

♦♦♦ Assess communication flow within each objective listed 

To thoroughly investigate and study the research objectives, the SLS 
Program Change Request (CR) change control process served as the surrogate for 
insight and data collection pertaining to decision making and communication 
patterns and processes. 
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CHAPTER II 


LITERATURE REVIEW 

2.1 Decision Definition 

What is a decision? The Merriam-Webster dictionary (2014) defines a 
decision as a determination arrived at after consideration. Mintzberg, et al. (1976) 
defined a decision as a commitment to a future action. Decisions are assumed to 
be clearly distinguishable and discrete events. Decision makers often can identify 
discrete decision points and feel a sense of completion at making a decision. 
However, decision boundaries are not always as clear as first thought [or 
assumed], and there is not always agreement on what events are involved in a 
given decision. Almost every decision involves a series of activities and choices 
nested in choices of wider scope, rather than a single simple choice (Poole & 
Hirokawa, 1996). 

2.1.1 Strategic Decision Definition 

One type of decision is a strategic decision. Strategic decisions express 
adaptation to opportunities, threats, constraints, and other characteristics of the 
environment (Papadakis, Lioukas, & Chambers, 1998). Strategic decisions have 
been described as committing substantial resources, setting precedents and 
creating waves of lesser decisions (Mintzberg, et al., 1976); as ill-structured, non- 
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routine and complex (Schwenk, 1988); and as substantial, unusual and all- 
pervading (Hickson, et ak, 1986). The significance of strategic decisions means 
that there is more at stake for those who stand to gain or lose from the decisions in 
tenns of material or reputational consequences (Child, Elbanna, & Rodrigues, 
2010). 

2.1.2 Strategic Decision Schemes 

Common decision making schemes with strategic decisions are consensus, 
majority vote, and decision by authority. Other methods include minority 
decision, where a subgroup decides, with or without the goodwill of the other 
group members; bargaining, arbitration, and compromise are also possibilities 
(Brilhart & Galanes, 1992; Gulley & Leathers, 1977; Jensen & Chilberg, 1991; 
Johnson & Johnson, 1987). When compared to uninstructed groups, consensus 
decision making produced better quality decisions but entailed more time (Hall & 
Watson, 1970; Nemiroff & King, 1975; Nemiroff et.al., 1976). Consensus is the 
best evidence of widespread commitment to a decision and without the 
commitment the decision has little chance of being effectively carried out. 
Moreover, the sometimes taxing process of working toward commitment yields 
not just commitment but better decisions (Nickols, 2005). 

2.1.3 Strategic Decision Impact 

By implication, strategic decisions are complex and involve a high degree 
of uncertainty (Mador, 2000). Strategic decisions, with important impact, attract 
the collective attention of more layers in an organizational hierarchy. This idea 
corroborated Dutton, et al. (1989), who argue that issues with great magnitude of 


8 



impact imply high interconnectedness with other relevant issues. Therefore, such 
issues attract more collective attention and thus result in higher hierarchical 
decentralization and lateral communication (Papadakis, Lioukas, & Chambers, 
1998). These results align with Dean and Sharfman (1993a) and Stein (1980), 
who suggest that the perceived magnitude of impact of a decision is among the 
strongest explanatory variables of decision making behavior, as decision makers 
act more comprehensive or rationally when a decision implies important 
consequences (Papadakis, Lioukas, & Chambers, 1998). 

2.2 Decision Dimensions 

As mentioned previously, a decision is a determination made after some 
consideration. Decision making is not a unitary event, but a complex social 
process involving the directing of attention, discovery, designing courses of 
action, evaluating alternatives and choosing among them (Simon 1965; Oliver & 
Roos, 2005). Two concepts - procedural rationality and politics - have clearly 
played central roles in the organizational decision making literature (Allison, 
1971; Carter 1971; Cyert & March, 1963; Eisenhardt & Bourgeois, 1988; 
Eisenhardt & Zbaraki, 1992; Fredrickson & Iaquinto, 1989; Hart, 1992; March & 
Simon, 1958; Mintzberg, Raisinghani & Theoret, 1976; Pettigrew, 1973; Pfeffer, 
1981; Dean & Sharfman, 1996). Substantial research by Dean and Sharfman 
(1993 a) has demonstrated that procedural rationality and politics are distinct 
dimensions of the strategic decision making process. 
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2.2.1 Procedural Rationality Dimension 

Procedural rationality is defined as the extent to which the decision 
process involves the collection of infonnation relevant to the decision, and the 
reliance upon analysis of this information in making the choice (Dean & 
Sharfman, 1993b). Managers who conduct and rely upon analysis in making their 
choices - those who use more rational strategic processes - will be more likely to 
develop effective plans for reconciling their organizations with environmental 
reality. As Bourgeois and Eisenhardt (1988, 827) put it, rational processes allow 
people to “form theories regarding which strategies will succeed.” Top managers 
who fail to systematically collect and analyze information about environmental 
trends and constraints will be much more likely to lead their organizations in 
nonviable strategic directions (Dean & Sharfman, 1996). 

2.2.2 Political Dimension 

Political behavior has long been recognized as an aspect of organizational 
decision making (Allison, 1971; Pettigrew, 1973). Since strategic decisions are 
made among people by people for people, they are a welter of action, interaction, 
and counteraction (Hickson, et al., 1986). An organization comprises distinct 
groups of people with different motivations for getting involved in decisions 
(Butler, 2002). The interplay of interests, conflict and power between individuals 
and groups means that the strategic decision making process can be characterized 
as political in nature (Wilson, 2003). 

Two key ideas underlie the political dimension of decision making. First, 
people in organizations have differences in interests resulting from functional, 


10 



hierarchical, professional, and personal factors (Hickson, et al, 1986; Pettigrew, 
1973). Second, people in organizations try to influence the outcomes of decisions 
so that their own interests will be served, and they do so by using a variety of 
political techniques (Pfeffer 1981; Dean & Sharfman, 1996). Hickson, et al. 
(1986) identify three main variables of political behavior or what they called 
‘politicality ’ : 1) Intervention or the extent of external influence, 2) Imbalance or 
the degree of uneven influence and 3) Contention of objectives or the extent of 
disagreement over objectives. Disagreement over objectives tends to reduce 
support for what has to be done later in implementation and diverts attention from 
exploiting knowledge about how to do it. Disagreement also contributes to 
unfavorable conditions (Dean & Sharfman, 1996; Sharfman & Dean, 1997; Nutt, 
1998; Eisenhardt & Bourgeois, 1989). The personality and style of the decision 
makers are important factors in the strategic decision making process. Some 
people have preference for data while others prefer to go with their gut; some are 
controlling, demanding and hoard information while others are inclusive, rely on 
and involve others. Conflict among the varying personalities and styles should be 
productively managed else conflict can be the detriment of the organization and 
the decision (Nickols, 2005). 

At first, Hickson, et al. (1986) argued that effective decisions must be 
based on organizational goals. However, Dean and Sharfman (1996) later 
assessed that political decision processes are typically not oriented toward 
organizational goals, are unlikely to produce complete and accurate information, 
and do not focus on environmental constraints. Consequently, they concluded 
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there exists a link between politics and unsuccessful decisions. Two years later, 
Papadakis, Lioukas, & Chambers (1998) conducted research that found with 
properly aligned tools, politics can have a positive influence on strategic decision 
making. They (1998) found that formal planning systems appear to have a 
positive influence on three aspects of the strategic decision making process: 
comprehensiveness or rationality, lateral communication, and politicization (i.e., 
politics). Results indicate that formal planning influences the way in which 
strategic decisions are taken and thus, to an extent, strategy itself. Indeed, by 
influencing comprehensiveness, lateral communication, and political activities, a 
formal planning system seems to act as a powerful input to the process of strategy 
making (Papadakis, Lioukas, & Chambers, 1998). This is in line with theoretical 
and normative speculations arguing that planning systems lead to more rational 
decision making (Armstrong, 1982; Langley, 1988; Papadakis, Lioukas, & 
Chambers, 1998), and results corroborate the prevailing view that fonnal planning 
systems encourage both lateral communication and political behavior (Langley, 
1988; Papadakis, Lioukas, & Chambers, 1998). 

2.2.3 Complexity Dimension 

An additional dimension of strategic decision processes not yet mentioned 
is complexity. Papadakis, Lioukas, & Chambers (1998) determined this 
dimensional factor associated more with the comprehensive or rationality aspect 
in their study than with lateral communication and politics. Astley, et al. (1982) 
argue that decision making may vary in tenns of complexity and cleavage. 
Complexity refers to the extent to which the topic (or decision to be made) is 
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intricate and may involve multiple considerations such as ambiguity, uncertainty, 
etc. and is likely to be greater in the case of innovative decisions. Cleavage, on 
the other hand (or politicality as it is labelled by Hickson, et al., 1986) involves 
the political dimension in decision making, for irrespective of complexity each 
topic is subject to the diverse (and often conflicting) view of various interests. 
Hickson, et al. (1986) also agree decision making may vary in terms of 
complexity. They suggest that decision complexity is caused by the extent to 
which the decision is unusual, the consequences that may stem from the decision, 
the extent to which it will set precedents for later decisions and the degree to 
which various interests and personnel become involved, both within and outside 
the organization (Hickson, et al., 1986; Rowe, 1989). 

2.2.4 Strategic Planning Tools 

Strategic planning models are designed to help organizations cope with 
rapid change to enhance an organization’s long-term prospects. Strategic 
planning anticipates new trends to which the organization must adapt (Jarboe, 
1996). Strategic planning models incorporate many of the elements of rational and 
creative models where rational models include data gathering, problem definition, 
solution generation, and solution evaluation and where creative models include 
components of classic, rational problem solving plus a social aspect with attention 
to arousing interest, motivation, belief in, and effort for the task (Jarboe, 1996). 
Strategic planning tools also place emphasis on assessment of the external and 
internal environments, increasing organizational learning, communication 
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between groups, and processes tailored to the organization as well as the situation 
at hand. (Jarboe, 1996). 

2.2.5 Section Summation 

A summation of the three dimensions of the strategic decision making 
process is illustrated in Figure 2.1. Each dimension has positive and negative 
attributes; however, through applying strategic planning tools, viable strategic 
decisions can be adequately detennined. Strategic planning tools can be the 
structure for who is participating and how, the process planning for data 
collection and analysis, the proactive, bidirectional communication of participants 
within organization, and/or the use of problem solving tools such as conceptual 
maps, creative decision analysis tools and techniques based on the complexity and 
severity of the problem (Mador, 2000). The development and application of 
formal planning tools provide positive influence on comprehensiveness, 
communication, and cleavage aspects of viable strategic decisions within the 
decision making process. 

Eisenhardt and Zbaracki (1992) say the strategic decision making process 
is best described as an ‘interweaving’ of both bounded rationality and political 
processes. More recent research emphasizes how executives make decisions 
using political processes in addition to rational procedures (Butler, 2002). 

Political behavior may shape the assumptions that feed into rational analysis, but 
rationality appears to be a superior dimension of strategic decision making 
because it alone, systematically and synthetically, leads to viable strategic choices 
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and an overall effective decision making process (Eisenhardt, 1989). This is 
reflected in Figure 2.1. 



Figure 2.1 Dimensions of Decision Making 


2.3 Decision Models 

Strategic decision making is complex involving many conflicting interests 
and environmental agents, taking long periods of time to make, and plagued by 
numerous interruptions, delays, disruptions, etc. Despite the complexity and 
seemingly random variations that characterize the strategic decision making 
process, there is evidence to suggest the process follows certain standard patterns 
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(Shrivastava, 1983). Shrivastava (1983) described these patterns by the following 
four strategic decision making models: 

> Managerial Autocracy Model 

> Systemic Bureaucracy Model 

> Adaptive Planning Model 

> Political Expediency Model 

Shrivastava’ s four models are described in terms of the six characteristics of 
decision making as defined by research studies of Shrivastava and Grant in 1982. 
The six characteristics are 1) problem familiarization and solution development, 

2) number and level of people involved, 3) motivation, 4) types of analysis, 5) 
role of organizational systems, and 6) environmental influences. These six 
characteristics are illustrated pictorially in Figure 2.2 and further expanded in 
Figure 2.3 (Shrivastava & Grant, 1982; Shrivastava, 1983). 
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Figure 2.2 Decision Making Characteristics 


2.3.1 Managerial Autocracy Model 

The first model is the Managerial Autocracy Model (MAM). Within the 
MAM, there is a single manager who is the key decision maker. A large amount 
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of power and authority rests with this single key manager who makes all strategic 
decisions himself with technical assistance from several subordinates. Decision 
making processes and outcomes are biased by the style and preferences of the 
decision maker in charge and not by system tools, procedures, or accumulated 
learning and experience of the organization (Shrivastava, 1983). 

2.3.2 Systemic Bureaucracy Model 

The second model is the Systemic Bureaucracy Model (SBM) where the 
decision making process is oriented toward systems and procedures rather than 
toward individuals and the individuals’ experience or expertise. Information 
flows in an erratic and impersonal manner with decisions made by using well- 
established norm, rules, and regulations. Within the SBM model, well-defined 
and documented stepwise procedures for handling all decisions are followed and 
as long as procedures are followed the organization is happy, despite the decision 
made (Shrivastava, 1983). 

2.3.3 Adaptive Planning Model 

The third model is the Adaptive Planning Model (APM). This model is a 
practical version of systematic planning for viable strategic decision solutions 
where plans are guidelines that are modified or deleted depending on the current 
analysis of issues. Problem formulation occurs at the time of development of an 
organizational plan that becomes the point of departure for strategic decision 
making. Qualified experts systematically evaluate the technical merits of the 
proposed alternatives in an effort to achieve efficient solutions to the problem 
(Shrivastava, 1983). 
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2.3.4 Political Expediency Model 

The final model is the Political Expediency Model (PEM). This model 
has several key decision makers or groups making decisions for personal gain, 
protecting or advancing their own interests even at the cost of organizational 
interests. While actual decision making is driven by interest group concerns, 
problem formulation and solution development can follow organizationally 
acceptable routines and procedures. Power is highly dispersed among 
organizational managers, and despite process and procedures existing, managers 
know how to circumvent the system to promote their own interests within the 
PEM model (Shrivastava, 1983). 

2.3.5 Section Summation 

A summation of how the six decision making characteristics described in 
2.3 correspond to each of the four decision models is listed in Figure 2.3 
(Shrivastava & Grant, 1982; Shrivastava, 1983). 


18 



CHARACTERISTICS 


MODELS 



MAM 

SBM 

APM 

PEM 

Problem Familiarization 
/ Solution Development 

• Restricted number of problem 
/ solution sets generated 

• One key manager develops 
solution with aid from 
assistants 

• Proliferation of problem / 
solution sets generated 

• Procedures for 
disseminating & 
communicating problem / 
solution sets are well 
developed & followed 

• Problem familiarization is 
almost nonexistent 

• Solution development 
revolves around 
modification of plans to 
accommodate changed 
conditions 

* Multiple problem / solution 
sets are generated but one 
set is championed by the 
vested interest group 

• Solution development is 
influenced by individuals or 
vested interests 

Numbers Level of 
People Involved 

• Usually one person 

• Decision made at top 
management level 

• Tasks forces / committees 
of technically qualified 
personnel are involved in 
the decision making 

• Decisions made at middle 
& top management levels 

• Several people form all 
levels of an org are involved 
in all phases of decision 
making 

• Decision made at middle or 
top management levels 

* A coalition of individuals or 
a single individual 

• Decision made at top 
management level 

Motivation 

• To improve efficiency 

• To have a result-oriented 
process 

• To satisfy procedural 
rationality 

• To fulfill written processes 
of organization 

• To fulfill plans 

• To implement action over 
making decision 

• To satisfy vested interests 

• To manipulate process to 
meet desires 

Analysis / Evaluation 

• Judgment or intuition of one 
key manager 

• Evaluation is extensive, 
detailed, objective 

• Cost benefit analysis is 
emphasized 

• Computational 

• Tech analysis and action 
planning are emphasized 

• Bargaining, negotiations, 
and group consensus on 
legitimate analysis to be 
done 

Role of Organization 

• Few systematic roles used 

• Decision making is highly 
personalized by key manager 

• Every activity is driven by 
bureaucracy 

• Communication and info 
sharing are predefined by 
org procedures 

* Strategic planning and 
learning tools play key role 

• Formal systems are side 
stepped 

• Personal knowledge of 
individuals widely used 

Environmental 

Influences 

• Very limited involvement of 
environmental agents 

• Environmental influences 
shape the problem and 
solution 

• Extensive negotiations with 
relevant environmental 
agents 

• Environmental agents may 
be brought in/accepted to 
join coalitions 


Figure 2.3 Summary of Decision Making Models (Shrivastava, 1983) 


Strategic decision making can vary among organizations; however, 
understanding these models, that Shrivastava showed evidence of standard 
patterns, can help decision makers redefine the process within the organization to 
make the process more rational and efficient by identifying and reducing the 
influences of undesirable, non-rational variables. For instance, one way to 
improve decision making is to involve systematic participation by relevant 
members who can handle technical complexity, risks, environmental constraints, 
and effectively communicate information in which to achieve viable strategic 
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decisions and implementation plans that yield solutions to problems (Shrivastava, 
1983). Understanding the decision making models can help organizations 
effectively execute the strategic decision making process and make successful, 
viable strategic decisions. 

2.4 Decision Determination 
2.4.1 Tracing a Decision 

Mintzberg and Waters (1990) quote a dissertation on decision making by 
Nicoladies (1960, 173): 

“It is evident on the basis of [my] analysis that an organizational 
decision is in reality a constellation or a galaxy of numerous individual 
decisions. Some of these decisions are “registered” in the book of the 
organizational activities, while others remain hidden in the inner sanctum 
of the human psyche. When and where a decision begins and ends is not 
always clear.” 

Decisions simply prove difficult to track down (Mintzberg & Waters, 
1990). Defining the beginning and the end of a decision process is also difficult. 
Does a decision begin when the group states a goal or problem, when it first 
becomes aware of an issue, or when a single member recognizes a need? Does it 
end when the choice is made, after the implementation period, or when the group 
explicitly takes up another issue (Poole & Hirokawa, 1996)? 

Do decisions precede actions? Can actions indeed trace back to the 
decisions made? Mintzberg and Waters (1990) argue that action can occur 
without commitment to act - as when a doctor strikes one’s knee - and challenge 
readers to consider the following comment by an executive of one of the world’s 
largest corporation in the 1990’s before thinking this is a far cry from the behavior 
of formal organizations: 
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“We use an iterative process to make a series of tentative decisions 
on the way we think the market will go. As we get more data we modify 
these continuously. It is often difficult to say who decided something and 

when - or even who originated a decision I frequently don’t know 

when a decision is made in General Motors. I don’t remember being in a 
committee meeting when things came to a vote. Usually someone will 
simply summarize a developing position. Everyone else either nods or 
states his particular terms of consensus” (Quinn, 1980, 134). 

Shift this into the more complex organizational setting where the 

commitment must be collective, and the problem of identifying decision 

magnifies enormously. Mintzberg and Waters (1990) say, given that an action 

was taken, and that broad support preceded it, the when and where consensus 

emerged must be found - for that must be the real ‘point’ of decision. In a 

bureaucratic society, citizens are expected to decide formally and receive 

approval before acting. However, not all organizations or all society are fully 

bureaucratic (Mintzberg & Waters, 1990). 

2.4.2. Strategy for Making a Decision 

So how are decisions made? Is there a particular flow or recurring 

strategy to decision making? Why are some decisions arrived at differently from 

others? Papadakis, Lioukas, and Chambers (1998) discovered the following 

strategic decision factors: perceived magnitude of impact, frequency, uncertainty, 

threat/crisis component and whether a decision emerges through discipline of the 

planning system of the firm, significantly influence the dimensions of the strategic 

decision making process, more than other environmental, organizational and 

managerial factors (Papadakis, Lioukas, & Chambers, 1998). Rowe (1989) 

argues that every decision is unique and can move along alternate routes 

depending upon the level of management involved and the stage at which they 
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become involved. A number of studies extend the argument further, contending 
that the role of ‘upper echelons’ or ‘top managers’ or ‘strategic leadership’ is 
important enough to determine strategy content and process (Child, 1972; 
Hambrick & Mason, 1984; Miller & Toulouse, 1986; Papadakis, Lioukas, & 
Chambers, 1998). Studies from Astley, et al. (1982) and Hickson, et al. (1986) 
became known as the Bradford Studies focusing on the ‘flow’ of management 
decision making. As referenced earlier, Astley, et al. (1982) proposed a model 
which argues that decision making may vary in tenns of complexity and cleavage 
(politics). They argue that particular combinations of problems and interests 
throw up particular processes and identify three main processes - sporadic, fluid 
and constricted - which can be linked with three types of subject matter - vortex, 
tractable and familiar - to form three ‘ideal type’ modes of decision making: 
vortex-sporadic, tractable-fluid and familiar-constricted (Hickson, et al., 1986; 
Rowe, 1989). 

Integrating research findings on decision strategies from Thompson and 
Tuden (1964) and Thompson (1967) with Astley, et al. (1982) research, the 
matrix in Figure 2.4 is formed (Dean & Sharfman, 1996). This matrix provides 
insight for how decisions are determined. 
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CLEAVAGE / POLITICAL 
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U) 

(Judgment) 
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X 

(Negotiation low, Scrutiny high) 

(Scrutiny high, Negotiation high) 


Figure 2.4 Decision Strategy Flow 

2.4.2. 1 Calculation Strategy 

If complexity of the decision to be made is low and the political behavior 
of group is low, then decision strategy is a simple computational (calculation) 
procedure. This is depicted within cell 1 of Figure 2.4. 

2.4.2.2 Inspiration Strategy 

Vortex-sporadic decision making mode (i.e., cell 4) is high in both 
complexity and politics. This mode sucks everyone into swirls of activity and is 
likely to be protracted, running into disrupting delays and impediments. 
Controversial and complex vortex matters tend to be processed in sporadic ways 
(Miller, 2010). Decisions are likely to be taken to the highest level where only 
inspiration by top management can provide an answer for what the decision will 
be (Hickson, et ah, 1986; Rowe, 1989). Thompson (1967) assesses that in cases 
of high uncertainty management acts in an inspirational manner by making 
obsolete any formal procedures and rules usually followed. 
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2.4.2.3 Judgment Strategy 

Tractable-fluid decision making mode (i.e., cell 3) is where less complex 
and least political tractable matters tend to be process in fluid ways (Miller, 

2010). If complexity surrounding a decision is high but exhibits low political 
agendas by the group, then the decision strategy is judgment when the group can 
collaborate as they scrutinize the details to reach understanding and an easily 
negotiated decision. The decision is typically non-controversial. The tractable- 
fluid decision making mode is more steadily paced that is formally channeled and 
speedy (Hickson, et ah, 1986; Rowe, 1989). 

2.4.2.4 Compromise Strategy 

Familiar-constricted decision making mode (i.e., cell 2) is where familiar 
matters which are the least complex and of mid-level politicality follow 
constricted ways (Miller, 2010). If complexity is low and politics is high, then a 
compromise strategy through high levels of negotiating may apply (Hickson, et 
al., 1986; Rowe, 1989). 

2.4.2.5 Decision Strategy Flow 

The model in Figure 2.4 is an abstraction, and it is not suggested that all 
decisions fit neatly into it, but rather, that decisions will approximate to the 
different cells. The decision outcome is dependent on the complexity of the 
problems and the politics of the interests and, to a lesser extent, the nature of the 
organization (Hickson, et al., 1986, Rowe, 1989). Management will, presumably, 
try to present as many decisions as possible as matters for calculation (i.e., cell 1 
of Figure 2.4), but if this is contested by the workforce or other managers, 
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allowing complexity and/or cleavage to increase, then decision strategies move to 
cells 2 and 3, or even to cell 4. The tension in the decision making process 
between cells 1 and 4 creates a major source of conflict within the organization 
and provides the political dimension (Rowe, 1989). 

2.4.3 Section Summation 

The aim of management therefore, is presumably to maximize the number 
of decisions where the outcome can (as far as possible) be accurately determined 
(i.e., calculation procedure in cell 1) and reduce dependency on compromise, 
judgment and, in particular, inspiration (Thompson & Tuden, 1964; Thompson, 
1967; Astley, et al., 1982; Hickson, et ah, 1986; Rowe, 1989). Management 
decision making may still be the ‘science of muddling through’ (Lindblom, 1959), 
but fonnal planning tool development may disentangle the muddle. The great 
appeal of the hybrid Thompson, Tuden and Astley matrix in Figure 2.4 is that it is 
both simple and fertile and a useful starting point for decision making analysis 
(Rowe, 1989). 

2.5 Decision Effectiveness 

Strategic decision effectiveness is defined as the extent to which a 
decision achieves the objectives established by management at the time it is made. 
Effectiveness as perceived by external constituencies may of course differ from 
management’s perceptions (Friedlander & Pickle, 1968). Managers have the 
capacity to influence organizational outcomes through strategic choice (Dean & 
Sharfman, 1996). Assumptions: 1) Decision processes are related to strategic 
choices, and 2) The relationship between strategic decision making processes and 
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effectiveness is that choices relate to outcomes. These two assumptions are 
plausible which suggests it is reasonable to expect strategic decision making 
processes to influence strategic decision effectiveness (Dean & Sharfman, 1996). 

2.5.1 Positive Influences 

Does the success or effectiveness of strategic decisions depend on the 
steps managers use to make them (Hitt & Tyler, 1991)? This question is 
fundamental to organization theory, as strategic decision making is a key element 
of management-centered conceptions of organizations (Astley & Van de Ven, 
1983; Dean & Sharfman, 1996). The concept of ‘strategic choice’ captures the 
extent to which the operating environment of the organization places limits 
around what managers can decide and how much autonomy they have in making 
those decisions (Child, 1972). Burgehnan (1991, 252) argued that “an 
atmosphere in which strategic ideas can be freely championed and fully contested 
by anyone with relevant information. . . .may be a key factor in. . .generating viable 
organizational strategies.” Pfeffer and Salancik (1978, 266) argued that, for a 
decision to be successful, “Information about the environment and possible 
consequences of alternative actions must be acquired and processed”. Therefore, 
in order for a decision process to result in an effective choice, it must be 1) 
oriented toward achieving appropriate organizational goals, 2) based on accurate 
information linking various alternatives to these goals, and 3) based on an 
appreciation and understanding of environmental constraints (Dean & Sharfman, 
1996). Results of Paul Nutt’s study (2005) of 376 strategic, non-routine decisions 
indicated that a rational, goal-oriented search approach tends to lead to more 
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successful outcomes. Hough and White (2003) in a simulated environment found 
a positive relationship between rationality and decision quality and that this 
relationship was contingent upon environmental dynamism (i.e., uncertainty) 
(Goll & Rasheed, 1997). 

The quality of information available to a group is one of the most 
important detenninants of successful group decision making (Kelley & Thibaut, 
1969). Nooraie (2008) found that rationality mediated the relationship between 
decision magnitude of impact and decision quality. Groups are more likely to 
reach high-quality decisions when their decision making processes are 
characterized by careful and painstaking examination and reexamination of the 
information on which the choice is to be based (Janis & Mann, 1977). Dean and 
Sharfman (1996, 389) note that “managers who collected information and used 
analytical techniques made decisions that were more effective than those who did 
not. Those who engaged in the use of power or pushed hidden agendas were less 
effective than those who did not.” They note that their study, despite using quite a 
different methodology, shows “that some of the findings of Eisenhardt and 
Bourgeois (1988) and Bourgeois and Eisenhardt (1988), extend beyond unstable 
environments to include stable ones as well” (Dean & Sharfman 1996, 389; 
Mador, 2000). Peter Senge (1990) observes groups are more likely to arrive at 
high-quality decisions when they employ a rational, as opposed to a political, 
logic in arriving at a final decision. 
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2.5.2 Negative Influences 

Crucial to successful decision making and problem solving in groups is 
the extent to which members’ interaction ensures that particular requirements of 
their tasks are being fulfilled. If they are not adequately addressed, the chances of 
the group’s making a good decision or identifying an effective solution to a 
problem are diminished (Gouran & Hirokawa, 1996). As Irving Janis (1989) so 
aptly noted, however, decision making and problem solving are activities that 
groups frequently perform under the influence of powerful social influences that 
can and do interfere with the ability of participants to satisfy the essential 
requirements of a decision making or problem solving task. Janis (1989) 
identified three sources of negative influences that he called ‘constraints’: 
cognitive, affiliative, and egocentric. When any of these three constraints become 
dominant, the interests of effective decision making are apt to be ill served unless 
action is taken. 

Janis (1989) relates cognitive constraints to perceived deficiencies in the 
resources (information, time, and skills necessary for perfonning the task) 
available to group members. When present, they lead to superficiality in the 
analysis of issues and alternatives a group may be considering. Affiliative 
constraints contribute to preoccupations with relationships and the well-being of 
the group (Janis, 1989). As a result, they can shift the focus of inquiry from 
making the best choice to the accommodation of differences in points of view. 
Janis (1989) also defined egocentric constraints as those deriving from the 
personal needs of the members (typically needs concerned with control). Such 
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constraints are productive of conflict, the culmination of which is often 
acquiescence rather than informed choice (Janis 1989; Gouran & Hirokawa, 

1996). If manipulation of data and secret communication exist, it could lead to 
selective and biased disclosure of relevant information (Pettigrew, 1973) and give 
rise to disappointing outcomes (Dean & Sharfman, 1996). 

2.5.3 Section Summation 

Although it is clear that group decision making performance is affected by 
a variety of factors and influences, there are many who suggest that the quality of 
communication that occurs as a group attempts to reach a collective decision may 
well be the single most important influence on the decision making success or 
failure (Hackman & Morris, 1975; Janis & Mann, 1977; McGrath, 1984; 
Hirokawa, Erbert, & Hurst, 1996). When a group communicates the right 
information at the right time in the right way, then the communication and the 
decision making will be more effective (Eikenberry, 2005). The existence of 
communication is necessary to measure the effectiveness of any decision (Hitt, 
Miller, & Colella, 2006). 

2.6 Communication 

Social systems theory considers communication as the basic element of 
any organization (Luhmann, 2000). Habermas (1998) finds that activities 
between people need a certain amount of communication, which must be fulfilled 
in order to coordinate actions effectively for the purpose of satisfying needs. In 
organizations the act of fulfilling these needs is an effective problem solving 
process, in other words, effective decision making (Habermas, 1998). Social 
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systems theory observes decision making as communication that opens future 
alternatives of action (Baraldi, 2013). 

2.6.1 Communication is Vital 

Communications is the instrument by which members of groups with 
varying degrees of success reach decisions and generate solutions to problems 
(Poole & Hirokawa, 1996). They found that communication constitutes decisions 
in at least two senses: 1) through communication the form and content of 
decisions are worked out, and 2) at a more fundamental level, decisions are social 
products embedded in “social reality”. Communication processes are the primary 
means through which social realities, as experienced by participants, are created 
and sustained, and therefore are the prerequisites for making decision. 

There are a variety of communication modes for decision making: face-to- 
face communication, written communication, audio communication, and 
electronically mediated audio/visual communication (Argyle & Cook, 1976; 
Hirokawa, Erbert, & Hurst, 1996). Results of investigations have been mixed, but 
the prevailing view is that less restrictive communicative modes tend to be 
associated with high-quality decisions than more restrictive modes (e.g., audio 
only) when more complex and difficult decision making tasks have been 
employed (Hiltz, Johnson, & Turoff, 1986; Hirokawa, Erbert, & Hurst, 1996). As 
the difficulty of a decision task increases, systematic face-to-face interaction tends 
to result in higher-quality outcomes (Hirokawa, 1988; Jarboe, 1988). Less 
restrictive mode, like face-to-face communication, provides group members with 
increased opportunity to exchange and utilize information in arriving at a solution 


30 



to the problem (McGrath, 1984). A study by Harper and Askling (1980) found 
that groups making high-quality decisions displayed higher-quality leadership, 
more open communication, and a higher proportion of active participants than did 
groups whose decision were judged to be of low quality. 

2.6.2 Communication, Process and Decision Making Performance 
Is group communication related to group decision making performance? Some 
contend that communication actually produces group decisions by creating and 
shaping the contexts with which those decisions are made and enacted (Hirokawa, 
Erbert, & Hurst, 1996). Hackman (1990) contends that group decision making 
performance is determined by three “enabling conditions”: sufficient group effort, 
adequate knowledge and skills possessed by group members, and appropriate 
performance strategies, processes, and procedures employed in reaching a 
decision. Moreover, he suggests that these enabling conditions exert positive 
influence on group perfonnance through the mediation of communication and 
interactions (Hackman, 1990). Habermas (1998) further surmises that social 
action and communicative practices are inherent parts of decision making process 
and perfonnance. 

Formal planning tools (i.e., planning models, procedures, building 
participant involvement, and promoting communication networks within the 
group) are social in intent with clear, task-related functions for the decision 
making process (Jarboe, 1996). The assumption behind involving people is that 
involvement increases the amount of information available to the group, increases 
commitment to the decision, improves dissemination of that decision, and 
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enhances commitment to group problem solving, thereby increasing the likelihood 
of quality thought throughout the process (Jarboe,1996). Procedures can also 
produce communicative behavior that produces outcomes (Jarboe, 1996). 

Outcome measures in decision making are often associated with primary 
task activity such as the quality of solutions, the number of ideas, or the 
uniqueness of ideas. Although these [quality, number and uniqueness] are the 
most practical ways to assess outcomes in laboratory research, there is little doubt 
that the value of a decision depends on the confluence of subsequent events 
(Reagan & Rohrbaugh, 1990). Reagan and Rohrbaugh (1990, 21) further offer 
“any assessment of the effectiveness of decision making performance requires 
directing primary attention to the process itself, not only to subsequent 
outcomes.” They studied various approaches to group decision making and 
discovered eight distinct criteria - adaptability, legitimacy, efficiency, goal 
centeredness, accountability, data based, participatory and supportability - to 
assess both process and decision effectiveness as communicated and conducted by 
a group (Reagan & Rohrbaugh, 1990). The eight criteria are summarized in the 
following questions by Reagan and Rohrbaugh (1990) as an evaluation of both 
decision making process and outcome effectiveness: 

> From a rational perspective, was the decision making process 
conducted by the group goal-centered and the resulting decision 
efficient? 

> From a political perspective, was the decision making process 
adaptable for the group and the resulting decision legitimate? 
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> From a consensual perspective, was the decision making process 
participatory by the group and the resulting decision supportable? 

> From an empirical perspective, was the decision making process 
data-based and the resulting decision accountable? 

2.6.3 Communicating Decisions 

A decision is the commitment to a particular course of action (Mintzberg, 
et ah, 1976), and the decision must be communicated clearly, coherently and 
convincingly (Nickols, 2005). One of the areas where the gap in decision making 
is widest is in communicating decisions. Decisions are made, but the 
communication of those decisions has shown to be ineffective or incomplete 
(Eikenberry, 2005). The following guidelines (Eikenberry, 2005; Busch 2012) 
help to successfully communicate decisions within an organization (and beyond): 

> Determine who (i.e., one executive, individual managers or both) 
and when (i.e., sooner the better) the decision is communicated 
and how (i.e., email, team meeting, voicemail, newsletter, etc.) 

> Clearly, correctly, and concisely communicate exactly what was 
decided and the rationale that led to the decision 

> Clearly stipulate how the decision’s effectiveness will be 
measured (i.e., communicating the measures for success is 
especially important to those who may disagree with the decision), 
and 

> Understand communication is a two way process (i.e., a complete 
communication plan of transmission of message, receipt of 
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message and feedback on message ensures individuals have 
received and understood the decision 
The results of a communication study by Habermas in 1998 show that 
both managers and employees found the best ways to communicate decisions 
were in meetings, email, or face-to-face conversation. Usually the channel [of 
communication] was chosen by the urgency of the message (Mykkanen, 2010); 
however, the important aspect is that the decision is communicated. 

2.6.4 Section Summation 

Substantial research has proven a direct link between communication, 
decision making performance through process, and decision making effectiveness. 
High quality decisions require communication for the adequate understanding of 
the problem, the formulation of viable strategic solutions to the problem and the 
effective implementation of those solutions to the problem. However, research in 
the processes of decision making and the communication of those decisions 
within the process is fairly new (Mykkanen, 2010). The outcomes of more 
research in this realm could further benefit practical applications of decision 
making and communicating decision within organizations, help organizations to 
evaluate whether the decision making process is too lengthy and whether the 
outcomes of the process reach the desired recipients, and help organizations 
concentrate on using more coherent information in decision making to improve 
organizational perfonnance (Mykkanen, 2010). 
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2.7 Literature Review Conclusions 


A resounding theme within the research literature is that strategic 
decisions are not routine, not well-structured, and not easily made due to a high 
degree of uncertainty associated with the problems to be solved, thus, rationality 
is necessary to achieve viable strategic decisions. Rationality is one of three 
dimensions associated with the strategic decision making process. Politics and 
complexity are also dimensions as presented previously in Figure 2.1 within 
section 2.2.5. Depending on the dimension evident, Figure 2. 1 illustrates a path 
as guidance for how teams can reach viable strategic decisions. Of the three 
dimensions, the application of procedural rationality fosters team consensus and 
produces effective and better quality decisions; however, this method can be time- 
consuming. Research shows that the optimal decision making process includes 
open and continual communication among all decision makers in repetitive 
evaluations of collected data to determine strategic decisions that are rationally 
assessed against environmental trends and threats. Greater the perceived impact or 
implied consequences of the decisions to be made, greater the need for decision 
makers to act rationally. Since people are involved, complexity and political 
influences are seldom absent from problem solving and decision making. To 
offset conflicts arising from the complexity and uncertainty amongst varying 
personalities and interests of the decision makers and amongst the varying details, 
risks, and consequences stemming from the problems to be solved, a team can 
apply fonnal assessment techniques such as procedures, devil’s advocacy, and 
dialectical inquiry that critique ideas or alternatives to drive out the best decisions. 
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Planning tools increase learning and communication within the team, enhance 
assessment of environmental constraints, allow processes to be tailored to specific 
organizational needs, and employ appropriate performance strategies to reach 
viable strategic decisions. Team interaction and open communication are vital to 
effective decision making where team involvement in making decisions increases 
not only the commitment to and quality of the resulting decisions, but also 
improves dissemination and execution of those resulting strategic decisions. 
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CHAPTER III 


PROBLEM DEFINITION AND STATEMENT 
3.1 Problem Definition 

The most important task in any systems decision process is to identify and 
understand the problem which is informed by understanding the concerns, 
objectives and constraints of the stakeholders (Parnell, et ah, 2011). Figure 3.1 
below shows the three components that help define the problem space for this 
study. They are 1) identifying the problem, 2) understanding the stakeholders 
who are the individuals possessing a vested interest in the problem, and 3) 
determining the best research approach to tackle the identified problem. Each 
component will be further discussed in the following sections. 



Figure 3.1 Definition of Problem Space (Parnell, et al., 2011) 
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3.1.1 Problem Identification 


At a high level, this study aimed at understanding the communication and 
decision making process as part of the Systems Engineering practice at the 
NASA/Marshall Center. The SLS Program, managed by the Marshall Center, 
served as the test bed for analysis. To that end, three SLS Change Requests (CRs) 
flowing through the Configuration Management change control process provided 
the specific basis for the research and analysis. 

The focus of this study was to understand the decision making process by 
tracking the three SLS CRs from initiation to official approval. To drive down to 
the crux of the problem for a thorough investigation and solution 
recommendation, the ‘5-Whys’ technique (Goodwin and Wright, 2012) was 
applied. 

1 . Why understand the SLS the decision making process? 

■f To understand how each CR was introduced, discussed, approved, and 
communicated within the decision making process 

2. Why understand how each CR was introduced, discussed, approved, and 
communicated within the decision making process? 

■f To understand key drivers leading to the decisions 

3. Why understand the key drivers leading to the decisions? 

■f To better understand if, how, and why the key drivers differed 
amongst the three CRs 

4. Why understand if, how, and why the key drivers differed? 
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•S Because the initial findings reflected that while all CRs were classified 
as Category 1 and successfully approved, each completed the process 
differently 

5. While each CR was successfully approved, why did each CR complete the 
process differently? 

•S Not sure. Thus, this was the identified root problem around which the 
research and analysis was focused to better understanding the SLS 
communication and decision making process. 

3,1.2 Stakeholder Analysis 

Stakeholders play an important role in the decision process for any project. 
When approaching a decision, stakeholders typically have their own schemata and 
filtering criteria in which to apply solutions that meet their intermediate needs. 
Schemata (or mental models) are cognitive structures that represent one’s general 
knowledge about a given concept or stimulus domain, including its attributes and 
the relations among those attributes (Oliver & Roos, 2005). Both the mental 
models and criterion are dependent on their disciplined area of expertise. 

With this study, the stakeholders were no different. Each stakeholder had 
a vested interest, be it power, legitimacy, or an urgency, to influence, not only the 
decision making process, but also the review and processing direction for each 
CR studied. This section will identify the stakeholders and the level of influence 
each contributed to the study. 


39 



3. 1.2.1 Stakeholder Identification 


For this study, the individuals who could provide insight into the technical 
discussion patterns, approval processes, resulting decisions, factors influencing 
the resulting decisions, communication of the resulting decisions, and 
effectiveness of resulting decisions (i.e., implementation success) for the SLS CR 
processing were identified as stakeholders and reflected here. 

1. Exploration Systems Development’s (ESD) Cross Programs 

o Orion / Multi-Purpose Crew Vehicle (MPCV) Program 
o Space Launch System (SLS) Program 

o Ground Systems Development and Operations (GSDO) Program 

2. SLS Program Office 

o SLS Program Managers 
o Stages Element Managers 
o Boosters Element Managers 
o Engines Element Managers 

o Spacecraft & Payload Integration Element Managers 
o Advanced Development Element Managers 

3. SLS Chief Engineers Office 

4. SLS Lead Systems Engineering Team 

5. SLS Discipline Lead Engineers 

6. SLS Element Discipline Lead Engineers 

7. Change Request Change Package Engineer 

8. SLS Configuration Management Office 
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o Configuration Management Manager 
o Configuration Management Release Desk 
3. 1.2.2 Stakeholder Salience Definitions 

To complete a stakeholder analysis for this study, the following definitions 
were employed: 

• Power - Made the final decision and responsible for overall budget and 
schedule resources 

• Legitimacy - Had direct relationship to decision to be made 

• Urgency - Had a critical need to find solution and/or critical claim to decision 
to be made 



TYPES 

0 - Non Stakeholder 

1 - Dormant 

2 - Discretionary 

3 - Demanding 

4 - Dominant 

5 - Dependent 

6 - Dangerous 

7 - Definitive 


Figure 3.2 Stakeholder Salience Types (Parnell, et ah, 2011) 
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Of the eight attributes classified for stakeholder saliency of influence 
(Matty, 2010) reflected above, there were three specific types fulfilled by the 
stakeholders within this study. Each of the three specific stakeholder type will be 
discussed in the following sections. 

3. 1.2.3 Definitive Stakeholders 

The Exploration Systems Development (ESD) division is responsible for 
ensuring technical, cost, and schedule details across three Programs align with 
agency, presidential, and legislative goals. The three Programs are reflected here 
in Figure 3.3. 
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Figure 3.3 SLS Program Definitive Stakeholders 


The Orion/MPCV Program, managed by Johnson Space Center (JSC), 
develops the crew exploration vehicle that will carry the crew to space, provide 
emergency abort capability, sustain the crew during space travel, and provide safe 
re-entry. The SLS Program, managed by Marshall Space Flight Center (MSFC), 
develops the heavy lift vehicle that will launch the crew vehicle, and eventually 
other modules and cargo for specified missions. The GSDO Program, managed by 
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Kennedy Space Center (KSC), provides the ground systems, infrastructure, and 
services to perform ground processing, launch and recovery, as applicable, for the 
SLS and MPCV Programs. The Cross Programs, collectively, comprise a 
definitive stakeholder exhibiting power, legitimacy and urgency. 

The SLS Program is a multi-element program which includes a Stages 
Element, Boosters Element, an Engines Element, a Spacecraft and Payload 
Integration Element and an Advanced Development Element as illustrated above 
in Figure 3.3. The SLS Program Manager leads the management, integration, and 
direction of all the SLS Element activities ensuring compliance and consistency 
with NASA Agency policy and priorities. All of the SLS Element Managers 
report to the SLS Program Manager regarding safety, schedule, performance, and 
cost details in the design and development of hardware and related systems of 
their respective Elements. The SLS Program Manger chairs the SLS Program 
Control Board (PCB) and is the decision authority for all SLS baseline changes. 
Consequently, The SLS Program Manager is a definitive stakeholder exhibiting 
power, legitimacy and urgency attributes. Specifically to the Change Request 
(CR) processing, the SLS Program Manager and SLS Element Managers depend 
on thorough reviews by subordinates/engineering discipline experts with 
concurrences and/or concerns with recommendations and forward plans presented 
at the PCB. 

With respect to this study, both the Cross Programs and SLS Program 
Manager were highly salient, major stakeholders with direct access to budgetary 
and other programmatic resources. They were powerful and legitimate 
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stakeholders who considered all technical recommendations, suggestions and 
options in relation to schedule and budget constraints to ultimately make the final 
decisions. 

3, 1.2.4 Dependent Stakeholders 

The majority of the dependent stakeholders discussed in this section are 
specific engineering entities within the SLS Program itself and illustrated in 
Figure 3.4. 
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Figure 3.4 SLS Program Dependent Stakeholders 


First, the SLS Chief Engineer (CE) executes Systems Engineering and 
Integration (SE&I) at the SLS Program level. The CE is responsible for the 
integrated SLS vehicle design and has a team of Chief Engineers distributed 
across the Elements. These Element Chief Engineers (ECEs) ensure the technical 
work at the Element levels meets the requirements of the integrated vehicle 
design. The CE chairs the Chief Engineer Control Board (CECB) which serves as 
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the engineering authority for SLS Program baselines. The CECB also functions 
as a technical pre-Board to the PCB where it reviews all changes within its 
defined authority and makes recommendations on any engineering and safety 
content seeking PCB final approval. 

Next is the Lead Systems Engineer (LSE) who also has a team of System 
Engineers distributed across the Elements. The LSE and the Element Lead 
Systems Engineers (ELSEs) ensure the planning and production of all multi- 
discipline deliverables for the SLS vehicle. The LSE also leads the change 
management effort within engineering for the SLS Program. 

The Discipline Lead Engineers (DLEs) are the single authoritative entity 
for understanding, assessment, and recommendations related to their assigned 
discipline for the entire vehicle. The DLEs are members of the CECB and are 
responsible for ensuring all integration with other disciplines and the Elements is 
achieved prior to seeking CE approval. DLEs are responsible for carrying any 
dissenting opinions to the CECB. 

The Element Discipline Lead Engineers (EDLEs) are technically 
accountable to the ECEs regarding data provided by the Elements for use at the 
vehicle level meets the needs of the vehicle and is technically adequate with 
respect to their discipline scope. 

The SLS CE, ECEs, LSE, ELSEs, DLEs and EDLEs were all dependent 
stakeholders. This group of moderately salient stakeholders had a direct 
relationship to the decisions under review. Overall, they had a critical need to 
seek solutions or had decisions made with respect to issues applicable to their 
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areas of technical/discipline expertise. The engineers had some power in 
providing technical suggestions and/or recommendations for consideration at a 
Control Board but do not make the final decision. 

Lastly of the dependent stakeholders is the CR Change Package Engineer 
(CPE); however, the CR CPE is not reflected on in Figure 3.3 because the CR 
CPE is an appointed position dependent on the technical expertise pertinent to a 
given proposed change. The CR CPE is responsible for the review and 
consolidation of comments from mandatory evaluators and the recommendation 
of a change disposition to the CECB and PCB. The CR CPE has a vested interest 
in the review of a CR and therefore categorized as a dependent stakeholder. As 
the CR shepherd, this moderately salient stakeholder has both a critical need and 
sense of urgency to obtain a thorough review, consensus, and a formal decision to 
the proposed technical change. 

While these dependent stakeholders (i.e., CR CPE, SLS CE, SLS LSE and 
the collage of expert engineers) did not make the final decisions, they were, 
however, the more deeply involved stakeholders in the CR initiation, review, 
approval, and implementation process. Furthermore, these dependent 
stakeholders were also the more influential stakeholders in technical assessments 
and decision package recommendations typically presented to the SLS Program 
Manager for final approval at the PCB. 

3. 1.2.5 Discretionary Stakeholders 

The SLS Receipt and Release Desk (R&RD) and Change Package 
Manager (CPM) are entities within the SLS Configuration Management (CM) 
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Office. The SLS R&RD is the authoritative point for all communication related 
to official CM products and serves as the official location for submittal of CRs. 
The CPM assists the CR CPE with the CR life-cycle process, from CR initiation 
to closure of the Control Board directive actions. 

From the study, the CM Office was a discretionary stakeholder. This 
latent salient stakeholder had no power or critical need for a decision to be made 
on the CRs; however, the CM Office did exhibit legitimacy by ensuring the 
official release of CRs for review and support with the CR processing, if/as 
needed. 

Figure 3.5 summarizes the saliency types of all the stakeholders associated 
with the decision making study. 



TYPES 

0 - Non Stakeholder 
1- Dormant 
2 - Discretionary 

3- Demanding 

4- Dominant 

5 - Dependent 

6 - Dangerous 

7 - Definitive 


Figure 3.5 Stakeholder Analysis for the SFS Decision Making Study 
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3.1.3 Research Approach 

For this study, a teaming approach was invoked to ensure the root problem 
was thoroughly researched, assessed, understood, and solved. The team was 
comprised of the thesis author and the SLS team members (i.e., stakeholders) who 
were personally involved in the reviews and decision making process for the three 
CRs. A survey was determined the better mechanism for data collection and 
analysis and thus employed. In addition to the survey, research and a better 
understanding of the formal SLS CM change control process was necessary to 
understand how and where the CR processing could have differed from the 
documented process. This entailed understanding the former CM change control 
process that included Tabletop reviews and the updated CM process that replaced 
Tabletop reviews with the Task Team review approach. 

To focus the research, any ‘known’, ‘partially known’ and ‘unknown’ 
details were determined. The ‘known’ details (or details with documented results) 
were: 

■ The Marshall Center had governing /guidance procedures/policies (NASA 
Procedural Requirements (NPRs), Marshall Procedural Requirements (MPRs), 
Handbooks, etc.) for program/project management and execution, safety and 
mission assurance, systems engineering, and technical design and standards. 

■ The SLS Program had a formal Configuration Management (CM) Plan that 
defined CM requirements, process and procedures for the control of SLS 
technical and programmatic documentation. 
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■ A CR, affecting a SLS baseline, followed the CM change control process for 
disposition and approval at the SLS PCB. 

■ All three CRs were classified as Category 1 which meant the CRs referenced 
the SLS Program baseline and required a rigorous control via an established 
and standardized CM process utilizing configuration control boards such the 
SLS PCB for official approval, control board directive actions, and 
concurrence sheets (SLS CM Plan, 2013). 

■ All three CRs were successfully approved and implemented, 
o CR53 

■ Originated: July 2012 

■ Approved: October 2012 
o CR70 

■ Originated: October 2012 

■ Put Back: January 2013 

■ Re-Released: February 2013 

■ Approved: April 2013 
o CR82 

■ Originated: October 2012 

■ Approved: December 2012 

■ All three CRs were processed and approved via different paths, 
o CR53 

■ No Tabletop reviews held 

■ No Form 4511 signed 
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Approving Board: PCB 


o CR70 

■ Series of Tabletop reviews held 

■ Processed through CECB and put-back at PCB 

■ CM process updated 

■ Task Team review held 

■ CR re-released 

■ Approving Board: PCB 
o CR82 

■ No Tabletop reviews 

■ Form 4511 signed by LSE only 

■ Approving Board: Joint PCB with JSC 

■ With all three CRs, discipline representatives were involved in the review 
process along with CM representatives. So technical expertise was available 
to aid CR technical processing, and CM expertise was available to aid CR 
change control processing. 

The ‘partially known’ details (or details with partial data available) were: 

■ With the time difference equaling six months to a year between CR review 
and survey input, the survey data provided by the stakeholders was at best 
memory recall with exact details not remembered very well. The survey 
respondents did their best to recall data and provide the best answer they 
could recall from memory. 

The ‘unknown’ details (or details certain to have no data or knowledge of) were: 
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■ The specific survey respondents’ names were unknown, and the linkages 
between the specific respondents and their survey inputs were unknown. 
Consequently, clarifying information could not be obtained after the survey 
responses were submitted. 

3. 1.3.1 Problem Statement 

The desired outcome of this study was to better understand the decision 
making and communication process within the SE practices at the Marshall 
Center and have an understanding of the level of its effectiveness. On the surface, 
there appeared from the root problem identified earlier (in 3. 1 . 1) to exist some 
potential inefficiencies. To accomplish the desired outcome, a teaming approach 
of the thesis author and participating SLS members through anonymous survey 
inputs collaborated using the SLS CR change control process as a surrogate to 
determine whether resource and communication efficiencies existed that would 
make the exercised decision making process more effective. A lesson learned 
from the Constellation reflected a less than stellar and ineffective strategic 
decision making process existed for the program. With respect to decision 
making and communication, this study investigated whether the SLS Program 
management had learned from Constellation and was sufficiently implementing 
recommendations from the abruptly cancelled program. 

3.1.3.2 Focused Research Questions 

To accomplish the objectives of the problem statement, the following 
research questions served as the specific investigation focus: 

■ How did the process differ for each CR? 
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■ Why did the process differ? 

■ What were the key drivers for the differences in CR processing? 

■ What were the benefits and/or drawbacks with the differences in the CR 
processing? 

■ Were the same resources expended for each CR or efficiently minimized? 

■ Did those involved in the CR processing feel their contribution was value 
added to the decision making and approval of the technical change? 

■ What strongly influenced the decisions? 

■ How were technical reviews, technical recommendations, resulting decisions 
and action plans communicated? Were they well-defined, well-structured, 
well-vetted and/or well-communicated? 
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CHAPTER IV 


METHODOLOGY 


A five step approach was formulated in developing the methodology to 
address the research questions for this study. The five steps were 1) detennine the 
strategic decision and communication specific characteristics of the CRs under 
investigation, 2) develop an instrument that will capture the necessary infonnation 
to better understand the decision and communication specific characteristics, 3) 
identify sample population for data collection, 4) administer data collection 
instrument, and 5) analyze collected data. 

4.1 CR Decision and Communication Characteristics 

Based on the literature review, the following list of characteristics were 
consistent with the strategic decision making and communication processes. As 
the first step of the study’s methodology, this list was evaluated for commonality 
with the three SLS CR decisions. 

■ Comprehensive with significant impact as a whole and on long-term 
performance 

■ Time intensive 

■ Significant commitment of resources 

■ Shared effort - not an isolated, unitary event 
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■ Complex social and communicative process 

■ Uncertainty 

■ Dynamically evolvable 

■ Solutions based on bounded rationality, insight, perceived magnitude of 
impact and inspiration 

■ Adaptable to opportunities, threats, constraints and environmental factors 

■ Influenced by politics 

■ Precedent setting 

■ Create waves of lesser decisions 

■ Non-routine and unusual 

Table 4.1 reflects the decisions for each of the three SLS CRs as well as 
the decision process flow for each CR. 

Table 4.1 SLS CR Decisions 


CR 

SLSL CR DECISION 

RESOURCES 

STAKEHOLDERS 

PROCESS FLOW 

DURATION 

CR53 

Implement Flight Termination System (FTS) 
architecture option 10A into the SLS vehicle baseline 

Cost 

Schedule 

Mass 

PM, CE, LSE, CSO, 
Stages, Booster & 
Payload Element 
Management, all DLEs 
all EDLEs, and 
GSDO & MPCV 
Programs 

Routine technical CR released 
per CE direction, no Table Top 
review, no signed Form 4511, 
CECB approval, and then PCB 
approval 

3 months 

CR70 

Update the Data Requirements List (DRL) with the 
latest Data Requirement Description (DRD) changes 
needed to reflect the baseline version. The update is 
required to support SLS Preliminary Design Review 
(PDR) planning. 

Cost 

Schedule 

Manpower 

PM, CE, LSE, CSO, 
all Elements, all DLEs, 
all EDLEs, and 
GSDO Program 

Routine programmatic CR 
released, 5 Table Top reviews, 
CECB approval, PCB puts 
back the CR due to cost 
impacts, CM process 
updated, Task Team reviews 
& signs Form 4511, CR re- 
released, CECB approval & 
then PCB approval 

6 months 

CR82 

Implement Core Stage Forward Skirt umbilical with the 
independent Vehicle Stabilization System (VSS) 

Cost 

Schedule 

Mass 

PM, CE, LSE, CSO, 
Core Element, 
Structure & 
Environments DLE, 
Test DLE, and 
GSDO Program 

Urgent technical CR released, 
no Table top review, Form 
4511 signed by LSE only, no 
CECB or PCB, approved @ 
JPCB 

2 months 


These CR decisions appear to be strategic, aligning with several of the 
characteristics listed above. Significant resources such as cost, schedule, 
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manpower, and vehicle mass were substantial evaluation factors. While the SLS 
PM was the final decision authority, the PM could not complete the engineering 
assessment alone. Oftentimes, the engineering change was complex requiring 
collaboration and technical expertise from many stakeholders. Table 4.1 
illustrated the breadth of technical expertise that was necessary for a 
comprehensive assessment of not only the feasibility of the technical change, but 
also of the impacts the change had on resources, vehicle design, mission success, 
safety, etc. This assessment plus the analysis of how to mitigate and/or manage 
risks appeared to be time consuming especially when the change entailed much 
uncertainty or ambiguity. While a shared and highly communicative effort among 
the organizational teams detennined decisions that were weighed against threats, 
constraints, and environmental factors, it undoubtedly also evoked political 
influences with potential negative contributions to the decision making process. 

To offset this, the team applied assessment techniques such as devil’s advocacy 
and dialectical inquiry that critiqued ideas or alternatives to drive out the best 
decisions. 

4.2 Research Method Development 

Step two was to develop an instrument to capture the necessary 
information to better understand the strategic decision and communication 
characteristics. This entailed detennining the best research method to collect the 
data and developing the mechanism to be thorough, succinct, and user-friendly. 
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4.2.1 Research Method Selection 


One method considered to obtain information was through face-to-face, 
one-on-one interviews. However, with hectic schedules filled with detailed 
technical design work on the SLS launch vehicle and individual Element (i.e., 
Boosters, Stages, Spacecraft/Payload, Engines) design reviews and deadlines 
looming, interviews were not a viable option. Consequently, the better approach 
to gathering data was a web-based survey. This type of survey afforded an 
anonymous, user-friendly environment for easy access and easy submittal for the 
participants. The research need, goals, and planned data collection method were 
presented to the SLS CE office and approved to proceed. 

4.2.2 Survey Instrument Development 

A software package called SurveyGizmo was first researched and found 
adequate for survey development, administration, data collection, and data 
analysis. SurveyGizmo also met the anonymous and user-friendly environment 
criteria. The goal was for a survey to be quick and easy for the respondents to 
complete, but also direct enough to evoke respondents to provide information of 
their perceived notions and understanding of the communication flow and 
decision making practices specifically for SLS engineering change request 
processing. 

The survey was designed to be succinct for respondents to give quick 
answers. Comment sections were also available allowing respondents to be as 
verbose as desired to provide any infonnation they wanted. 
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Construction of the specific survey questions centered on the existing 
philosophy of the SLS CR change control process. Three primary areas of 
exploration (i.e., process, problem identification, and success) were defined 
within the survey. The breakdown of the different type of questions per area are 
listed in the next sections. 

4.2.2. 1 Process Oriented Questions 

These survey questions were focused on the mechanics of the SLS CR 
change control processing. In theory, the CR was generated by an entity (i.e., 
person or group) that requested a technical or programmatic change and then 
shepherded the CR through a series of events (i.e., official release of CR, formal 
review period for comments to the CR, discussions of comment dispositions 
and/or potential impacts, and notification of CR approval). The intent of the 
survey questions was to gain insight into the respondents’ knowledge of the CR 
change process and to detennine if the process was effectively practiced. 

• Who (person or group) generated the CR? 

• How were you notified of the CR for assessment? 

• Were you involved in the discussion of any impacts stemming from the 
dis-positioned comments from the review of the CR? Explain. 

• How were you notified of the Table Top or Task Team Review for this 
CR? 

• When did the decision / approval of the CR officially get to you 
(approximate month and year) or Never? 
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4.2.2.2 Problem Identification Questions 

The problem identification questions focused on identifying gaps or issues 
(underlying or blatant) which the respondents personally experienced or 
witnessed during the CR decision formulation. The intent was to not only 
pinpoint and understand the problem, but also to assess for potential 
recommendation for improvement or efficiency to SLS Program management 
and/or SE&I technical authority. 

• What was your motivation in reviewing the CR? 

• Would the assessment have benefited from additional expertise or input? 

If yes, what expertise would have improved the CR assessment? 

• Do you feel you had adequate time and/or CR related materials to perform 
an assessment of this CR? If no, what hindered your review? 

• Do you think your comments were assessed adequately in the Table Top 
or Task Team Review process? Explain. 

• If you did not fully understand the CR decision and its implementation 
plan, what would have aided your understanding? 

• Were there any gaps in communication during the CR review? If yes, 
what were they? 

• Were there cost or schedule impacts due to communication glitches during 
the CR review? If yes, what were they? 

4.2.2.3 Success Oriented Questions 

The success oriented questions focused on identifying successes and/or 
positive tenets of the decision making and communication process the 
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respondents experienced or witnessed. It was important that the active 
participants in the CR process shared technical expertise for strategic decision 
making. They also needed to feel their concerns were heard and that they were 
viewed as a knowledgeable contributing member of the team. The intent of these 
questions was to determine the respondents’ level of involvement in decision 
making, detennine the level of awareness and agreement with the resulting 
decision, and to understand the respondents’ perspective of the decision making 
process effectiveness. 

• Do you believe you provided a needed contribution to the CR assessment? 

• How do you feel your comments were received and dis-positioned? 

• Do you think your comments were assessed adequately in the Table Top 
or Task Team Review process? Explain. 

• Did you fully understand the CR decision and its implementation plan? 

• To what degree did you agree with the CR decision? 

• From your perspective, were your concerns with the CR dealt with 
effectively? 

• From your perspective, how would you describe the decision process with 
respect to the CR? 

Each of the three CR surveys contained these same types of questions with the 
header of the survey serving as the distinguishing factor between them. The 
survey content was vetted through SLS and approved. 
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4.3 Sample Population Identification 

Step three was to identify the population of individuals most involved with 
each CR’s decision making and communication process. The technical scope, 
urgency, resources, and population varied for each CR. Participant data were 
collected from two primary sources: 1) the original email notification of the CR 
issuance from the SLS CM Release Desk, and 2) the consolidated matrices of 
reviewer comments for each CR. The official notification of a CR’s release came 
in the form of an email from the SLS CM office with a pre-coordinated, pre- 
approved distribution list of multi-layer managers (LSE, CSO, ECEs, and DLEs). 
These managers participated in the CR change control process, and they could 
notify members of their teams either verbally or by the forwarded SLS CM email, 
requesting their participation in the CR review also. Any one that provided 
comments to the CR CPE were captured in a consolidated matrix for formal 
review and disposition. Therefore, these two primary sources were selected for a 
comprehensive list of participants for each CR. The list of names compiled for 
each CR was vetted through SLS technical authority management and approved. 
This list of names served as the population for each CR and ranged from 13 
individuals to 118. 

4.4 Survey Administration 

With an approved research request, survey, and population from the SLS 
Program, the next step was to administer the surveys. Official notification of the 
survey originated as an email from the SLS CE Office and included the secured 
participant distribution list and a link to the web-based CR survey. The 
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participants were encouraged to support and provide as valid and candid 
information as possible. 

4.5 Data Collection 

All data were collected in one round of structured, anonymous surveys 
with SLS engineers, managers, and administrative support who were actively 
involved in making the decisions under study. The survey statistics for each CR 
were: 

• CR 53 - Flight Tennination System (FTS) Architecture Option 10A 

o Survey Population: 44 
o Survey Responses: 8 (per SurveyGizmo) 
o Survey Success: 18% 

• CR 70 - Data Requirements List Update 

o Survey population: 118 
o Survey responses: 38 (per SurveyGizmo) 
o Survey Success: 32% 

• CR 82 - Core Stage Forward Skirt Umbilical 

o Survey population: 13 
o Survey responses: 5 (per SurveyGizmo) 

■ 6 submits of survey 

■ 1 removal of indeterminate submittaU 
o Survey Success: 38% 

+ The data in this survey submittal could not be rationalized. Vague and 
conflicting answers were consistently provided throughout the survey input. With 
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an anonymous survey, an inability to contact the individual for clarification, and 
an inability to rationalize the contradicting answers, the decision was made to 
remove the indeterminate submittal from analysis. 

4.6 Human Subject Testing 

To ensure the ethical treatment of human subjects, the following 
conditions were met within this study: 1) anonymity was stated and guaranteed 
for the participants, 2) data would be reported collectively per CR, 3) participating 
organization gave consent, and 4) every participant was over 19 years of age. 

Due to an oversight, the Institute Review Board (IRB) approval was not pursued. 
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CHAPTER V 


DATA RESULTS AND ANALYSIS 


Data results for the three areas of exploration (i.e., process, problem 
identification, and success) are presented and analyzed within this chapter. First, 
observed discrepancies in the data and how those discrepancies were handled will 
be explained followed by the presentation of the survey data results and analyses. 
A summary of the main findings and key decision drivers determined from the 
study conclude the chapter. 

5.1 Observed Data Discrepancies 

Two data discrepancies were observed in the survey results. Each of these 
discrepancies will be briefly explained to provide a better understanding of how 
the discrepancy data was assessed within the data analyses presented and 
discussed throughout this chapter. 

5.1.1 ‘Other’ Category Discrepancy 

When the survey was originally launched, a glitch was discovered where 
respondents could not submit the survey if a couple of questions were left blank. 
The issue was quickly rectified, and the survey re-launched. However, in a few 
occurrences of the early data submittals, the ‘Other’ data field captured benign 
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verbiage, such as “The survey required a response in this block.” as one example. 
This data was compiled and included in the complete data set of survey results in 
this study; however, the erroneous data did not carry any weight or provide any 
value in the final data analysis of the results for the overall study. Consequently, 
if there was no justification given or additional explanation for the ‘Other’ 
category of responses in this data analysis write-up, it was because it was related 
to the few occurrences of these benign answers received. 

5.1.2 Percentages >100% Discrepancy 

The respondents were asked to “Check One” for an answer to the 
questions in the survey; however, the respondents would, in a few occurrences, 
check two or three answers for a question. An example of such an occurrence 
was with survey question: How were you notified of this CR for assessment? 
Options available were “Direct email from CM”, “Email from DLE/EDLE”, 
“Verbal from DLE/EDLE”, and “Other”. The respondents were asked to select 
one; however, a few respondents checked multiple answers. Furthermore, 
SurveyGizmo would count the number of answers given for each answer and 
divide that by the total number of respondents for the question. Consequently, in 
these instances of multiple answers given, the total percentage resulted in a value 
greater than 100. Nine out of 27 questions across all three CRs resulted in 
multiple answers given and percentages exceeding 100%. In each case, the 
percentages were normalized for analysis and were denoted with an asterisk in the 
title of the data results within this chapter. 
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5.2 Definition of 75% Delineator 


Poole and Van de Ven (2010) sunnised that process research is all about 
finding temporal patterns, and the forms of the representation contributes 
significantly to pattern recognition. In an empirical study of decision making 
processes, Poole and Roth (1989) developed a three-tier, phasic timeline that 
tracks the task functions and the working relationships of a group to the 
percentage of the discussion the group has at various phases of decision making. 
They determined that the 75% marking on the timeline denoted the optimal level 
of participation and communication amongst the decision makers for the 
formulation and realization of strategic decision solutions. Consequently, a value 
of 75% was selected as a delineator in the assessment of process effectiveness. 
This 75% delineator was utilized in the evaluation of the frequency that the 
desired selectable answers to the survey questions were chosen by the CR 
respondents. The reasoning was the more often a desired answers was selected, 
the more effective the respondents found the process to be. 

5.3 Process Results and Analysis 

There were nine process-oriented questions pertaining to the three CRs. 
The results will be discussed in the following sections. 

5.3.1 Results per Question 

Ql: Who (person or group) generated the CR? 

• CR 53 
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• 88% (representing 7 out 8 respondents) answered a combination of 
EV72 and/or Flight System Safety Working Group which was 
consistent with the originating organization listed on the CR 

• 22% of respondents answered with a broader entity such as System 
Engineering / SLS Systems Engineering & Integration (SE&I) 

• CR 70 

• 47% (18 out of 38 respondents) answered a combination of EE 12 
and/or Configuration Management Office which was consistent with 
the originating organization listed on the CR 

• 1 1% (4 of 38 respondents) identified other organizations they thought 
were the originator 

• QD02 

• Change Package Manager 

• 8% of the respondents could not recall an originator 

• Do not know 

• Don’t have time to go look this up 

• Unknown 

• Lastly, like earlier with CR53, 34% of the respondents (13 of the 38) 
answered with a broader entity such as SLS System Engineering as the 
originator 

• Level II SE&I 

• SLS SEI Management 

• CR 82 
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• 80% (4 out of 5 respondents) answered a combination of EV74 and/or 
SLS Vehicle Integration which was consistent with the originator 
organization listed on the CR 

• 20% representing a single respondent thought the Core Stage element 
originated the change request 

Across all three CRs under study, an average of 72% of the respondents had a 
decent understanding and awareness of what organization and/or individual 
originated the change request and to whom they provided comments. 

Q2: How were you notified of this CR for assessment? 



An average of 67% of the respondents indicated they were identified as 
mandatory reviewers of the change request via the original distribution email 
from the SLS CM Release Desk. This initial contact from the SLS CM Release 
Desk was typically and appropriately formal. This percentage indicated the right 
technical disciplines were researched and identified early for review of the 
engineering change. An additional 12% were notified less formally via email or 
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verbal request from a manager-type to review the change request. The final 21% 
were notified by ‘Other’ means such as: 

• Email from DLE agent 

• Verbal from a LSE 

• Email from office SLS and/or MPCV CM support 

• Verbal from a FTS Trade Team representative 

• Not notified 

Q5: Do you believe you provided a needed contribution to the CR assessment? 



Figure 5.2 Summation of SLS CR Survey Question 5 Results* 


Considering all three CRs, 75-90% of respondents felt they provided a necessary 
and beneficial contribution to the CR reviews. 

Q6: Would the assessment have benefited from additional expertise or input? 
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Figure 5.3 Summation of SLS CR Survey Question 6 Results* 


By the input of “no additional expertise needed” from an average of 88% of the 
survey respondents, it appeared the process proved effective in including the right 
expertise to review, rationalize, and discuss impacts, risks, and workable solutions 
of the CRs. For the other 12%, most of the comments received suggested that 
lack of resources (time, budget) were the reason additional expertise would have 
helped. 

Qll: Were you involved in the discussion of any impacts stemming from the dis- 
positioned comments from the review of the CR? If not, please explain. 
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80% 
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Figure 5.4 Summation of SLS CR Survey Question 1 1 Results* 
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The majority (>80%) of the respondents were actively involved, formally or 
informally, in the CR decision assessments and deliberations regarding the 
magnitude of risks and consequences of those risks. While most of the 
respondents were involved in potential impact discussions and resolution plans, 
there were a minority of respondents who were not involved. The following 
question provided insight into respondent exclusion from potential impact 
discussions. 

Q12: Why were respondents not involved in impact discussions? 

• Reviewer oversight or Reviewer unaware of impacts or discussions regarding 
them 

o Without my knowledge, a decision was made to eliminate a [Data 
Requirements Description/Definition] DRD that affected the document 
within my responsibility. 

o Email communication [was] used for dispositions. Minimal info as to 
impacts and little info as to big picture effects. 

o Just focused on the disposition of my comments via email. I was not 
involved in any group discussion. 

• Either had no comments or had non-trivial editorial comments 

o I did not have any comments nor did anyone from my organization. 
CR53 was a well-vetted change prior to the CR release. 

o Comment was "editorial", no discussion required. 

• Reviewer only involved at CECB level 
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o Outside of the CR presentation to the CECB, I was not involved in any 
discussion of comments. 

Q13: If you were involved in impact discussions, how were you notified of the 
Table Top or Task Team Review for this CR? 
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Figure 5.5 Summation of SLS CR Survey Question 13 Results* 


Verbal and/or written (i.e., email) notifications from the CR CPE and/or SLS 
discipline management were the dominate methods practiced when notifying the 
respondents of CR discussions regarding impacts, consequences, and 
workarounds. The notification and communication at this stage in the process 
was more informal and appeared appropriate. An average of 12% of respondents 
was notified of CR discussions by other methods as described in the follow-on 
question. 

Q14: What were ‘Other’ means of review notification? 

• Email from task team representative from within home organization 

• I was involved in control board reviews, not table tops. 
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• I was notified by the document owner, LSE and through the formal Booster 
CM process. 

• I was involved in a division assessment of the impact. The division 
management assessed the scope and provided a cost impact. Coordinated with 
my management and not directly to the CR. 

Q17: When did the decision / approval of the CR officially get to you 
(approximate month and year) or Never? 
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Figure 5.6 Summation of SLS CR Survey Question 17 Results 


It appears all respondents were aware of the approval of the CRs with the 
exception of 6 respondents (i.e., 16%) on CR70 who were never notified of the 
decision or approval of CR70. With respect to the calendar timeframe inputs, the 
accuracy in answering was dependent on the respondents’ memory. The inputs 
ranged over 2-6 month intervals for each CR. One thought on the variance in the 
timeframe answers was that the respondents may have been involved in the PCB 
where the CR was final approved and gave that date as a reference. Other 
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respondents may have given another date based on the official receipt of the final 
approved document with all changes and decisions incorporated. Depending on 
the magnitude of comments, the document update and release could have taken a 
few months post PCB approval. 

5.3.2 CR Process Analysis 

All the answers submitted by the survey respondents with respect to the 
process-oriented questions have been detailed earlier in this chapter. This section 
will attempt to detennine the effectiveness of the CR change control process by 
evaluating the desired answers selectable for the process-oriented questions 
against the survey results from the respondents’ answers across all three CRs. 
When analyzing the change control process, the desired choices were identified as 
receiving official notifications of the change and of meetings and/or discussions 
about the change and possessing a good understanding of who initiated the 
change, its purpose, reviewer expectations, and time constraints for review, 
decision formulation, and CR approval. Table 5.1 lists these desired answers and 
the resulting percentages for each CR. 


Table 5.1 Desired Selectable Answers for Process 


Desired Selectable Answers 

CR53 

CR70 

CR82 

Received official CM notification email 

60% 

57% 

83% 

Understood who originated the CR and why 

88% 

47% 

80% 

Provided a needed contribution to the CR decision 

75% 

89% 

80% 

Felt adequately involved in impact discussions 

76% 

78% 

100% 

Felt adequately involved in formal reviews 

71% 

68% 

84% 

Knew when the CR was approved 

88% 

73% 

100% 
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The CR results for the desired process related questions ranged from a 
lowest value of 47% to a highest value of 100%. Evaluation of the three CR 
results with a 75% delineator revealed that the CR82 respondents observed no 
process inefficiencies, the CR53 respondents observed a few, and the CR70 
respondents observed the most inefficiencies in the processing of the DRL update. 
The CR53 results ranged from 60% to 88% for the desired process-oriented 
questions with results falling below the 75% delineator in the notification of the 
vehicle flight software architecture change and Table Top Reviews. The CR70 
process related results ranged from 47% to 89% with percentages falling below 
75% in understanding the CR origination and purpose as well as in the 
notification of the CR, team reviews, and resulting CR approval. Improvements 
within the communication realm of the process are suggested, especially when 
16% of the CR70 respondents were unaware of the official approval of the CR on 
which they participated. The results reflecting CR70 as having the most process 
inefficiencies appear logical since the baseline SLS CR change control process 
was revamped during that CR review. 

5.4 Problem Identification Results and Analysis 

There were eleven problem identification questions pertaining to the three 
CRs. The results will be discussed in the following sections. 

5.4.1 Results per Question 

Q4: What was your motivation in reviewing the CR? 

• CR53 

o To establish a FTS architecture for SLS 
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■ Participated in the decisional process for the FTS architecture, 
which resulted in having this CR be generated 

■ Gain knowledge of the Flight Tennination System (FTS) 
o Ensure consistency among Cross-Programs and other activities 

■ Making sure it has the appropriate considerations relative to 
ground operation activities 

■ To assess that the change was consistent with trade study 
results and an agreed to option 

■ Understand if impact to crew vehicle was acceptable 

■ Ensure the specific Flight Tennination Architecture definition 
that is required to be represented in the SLS technical baseline 
is documented in the appropriate interface control document. 

o Safety 


■ Concern that FTS implementation could lead to a safety risk 
and a future redesign to mitigate the safety risk. Architecture 
seemed to be driven by a view to simplify a design (i.e., 
remove components) rather than a full integrated stack systems 
view. 


• CR70 

o Mandatory Evaluator 

■ DLE role 

■ It's part of my job 

o Accuracy of technical / discipline-specific data in DRL 
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■ Responsible for the development of several documents affected 
by this CR / Motivated to ensure the accuracy of the data in the 
DRL / This document affects my work / To ensure that my 
input to the document was correctly implemented / Ensure the 
documents under my functional team were accurately captured 
in terms of content and delivery milestones. 

■ We needed to review the CR to ensure that our organizations 
[Data Requirements Description/Definition] DRDs were 
correctly represented and that there were no impacts to our 
organization relative to other DRDs 

■ The DRL is like the Rosetta Stone for all the pertinent 
information regarding products generated by one discipline and 
products generated by other disciplines in which there are 
many stakeholders. Need to assure program integration 
between the vehicle and ground services occurs smoothly / 
Reviewing content relative to ground operations deliverables / 
Having the necessary documentation properly identified and 
baselined. 

■ CR was related to the SLS DRL which directly effects the 
SPIO Element by defining data products required from payload 
integration team. 

■ Review for correctness from the standpoint of Integrated 
Avionics and Software (IAS) 
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o Align with Agency, Center and Cross Program requirements 


■ Ensure the program had the correct deliverables identified in 
order to comply with Agency and Center requirements for the 
development of the program. 

■ This was a Cross Program change request that could impact 
another Program or ESD 

■ To ensure DRL would adequately address changes to category 
1 and 2 documentation associated with verification and 
validation needs across all programs. 

o Deliverables alignment and document flow to external entities 

■ It directly impacted deliverables from the element office(s) and 
external entities with potential for cost impact / Assure 
alignment between L2 expectations for data deliveries from the 
elements / Interested from perspective of proper classification 
of documents for proper flow down to external entities 

• CR82 

o Assigned as a mandatory reviewer 

■ Part of the job 

o Accuracy of technical / discipline-specific data 

■ Document owner 

o Ensure consistency among Cross-Programs and other activities 

■ Minimize operational and interface impacts between the 
vehicle and ground operations 


77 



Q6: Would the assessment have benefited from additional expertise or input? 
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Figure 5.7 Summation of SLS CR Survey Question 6 Results* 

Between 80 and 100% of the respondents believed the technical assessment they 
provided was sufficient. 

Q7: If yes, what expertise would have improved the CR assessment? 

• Objective independent assessment presented to the Cross-Program tech 
authorities 

• Vice identifying specific expertise, respondents tried to justify why certain 
expertise was not included 

o Unclear scope 

■ Unsure if individual knew if all documents were necessary [on 
CR 70] 

o Limited resources 

■ There are limited amount of resources to review all CRs, each 
organization tries to place those with the most expertise on the 
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CR; however, there is a lack of planning of CRs to ensure that 
SMEs are not overburdened with CR reviews. (CR70) 

■ Additional expertise is always needed but that need has to be 
balanced with the cost of managing excessive input. I believe 
the balance achieved for this revision was reasonable. (CR 70) 
Q8: Do you feel you had adequate time and/or readily available CR related 
materials to perform an assessment of this CR? 
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Figure 5.8 Summation of SLS CR Survey Question 8 Results 


The data showed that the majority of the respondents felt they had adequate time 
and resources available for the review of each CR; however, there were a few 
concerns as explained in the following question. 

Q9: If no, what hindered your review? 

• Unclear change purpose or intent 

o CR 82 was an urgent CR when it came out. The 

information/background about it was very confusing. Real 
understanding of the thrust arrived when CR was explained at the 
board. 


79 


Insufficient time / Workload 


o Needed more time to fully review this document 
o Heavy review load including other CRs and documents 
o While the review period for this CR (70) was sufficient for one CR of 
this magnitude, this CR was not the only one under review. That is the 
price of a tight schedule; multiple changes being reviewed 
simultaneously by the same experts. We do the best we can, and 
request extensions when we feel we cannot accomplish an appropriate 
review. But there's never enough time to do it all. 
o Preparation for internal milestone review. CM system had a number of 
changes in the system to review. 

Q15: Do you think your comments were assessed adequately in the Table Top or 
Task Team Review process? 
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Figure 5.9 Summation of SLS CR Survey Question 15 Results* 


The data showed the majority of the respondents across all 3 CRs thought their 
comments were adequately assessed. The explanation of the 33% dissension was 
explained in the follow-on question. 
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Q16: If no, please explain. 


• The future implications [of FTS option 10A] were not given much 
consideration. The future configurations [were] not viewed as design 
considerations. 


o I felt the solution was workable but we did not adequately consider 
future implications or risk in operational scenarios. 

Q18: Did you fully understand the CR decision and its implementation plan? If 


not, what would have aided your understanding? 
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Figure 5.10 Summation of SLS CR Survey Question 18 Results 


Across each of the 3 CRs, 80-100% of the respondents understood the resulting 
decisions and the implementation plans for the decisions. Issues that impacted 
understanding were listed in the next question. 

Q19: If not, what would have aided your understanding? 

• A better big picture understanding of intent of CR [70] and what problems it 
was solving. 

• We were plowing new ground with what it meant to execute a hard T-0 design 
— that design is still in work after all these months. (CR 82) 
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Q22: Were there any gaps in communication during the CR review? 
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Figure 5.11 Summation of SLS CR Survey Question 22 Results 


On average, 78% of the respondents experienced no communication issues, 13% 
experienced minor communication gaps, and 9% of the respondents saw major 
issues in communication. The next question addressed the major and minor 
communication issues identified by 22% of the survey respondents. 

Q23: If yes, what were they? 

• Unclear cost impacts 

o Lack of understanding life cycle cost impacts 
o Cost assessment was not performed until the CR was out for formal 
review. Several significant cost impacts had to be worked out through 
the board process. 

o Some entities "piled on" with cost impacts at the PCB meeting instead 
of writing the cost impacts down formally through comments. 
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o Unclear cost impacts from some parties in the days leading up to the 
boards which resulted in "piling on" a bit at the board meetings. This 
was minor, though. 

Unclear deliverables schedule 

o Updates required inputs to be scheduled per the SLS-SCHE-164 
[document] which did not accompany the CR. No communication as 
to how the scheduling of deliveries would be handled. 

Unclear SLS Task Team Review Process 

o The SLS Task Team review is a bit confusing to most outside 

organizations - who participates, how are they chosen, how are they 
notified. Also sometimes a gap in closing the loop with CR 
reviewers/commenters 

Unclear content / scope of change 

o This CR [70] was release in 2012 and then re-released in 2013 as Rl. 
There was some question as to what was retained in the comments 
from R0 review to Rl - just caused additional review of the Rl 
version. 

Unrealistic review process for substantial changes 

o The method of doing changes of this importance and magnitude is 
broken. The DRL is a document that should be given mandated 
undivided attention though it is just one of many priorities when it is 
worked as part of an existing program. The magnitude of the change 



and the method of review of the changes and inputs from the various 
commenters created a communication nightmare. 

• Unclear purpose 

o [CR 82] was an urgent CR whose intent was not well communicated. 
Couldn’t figure out the thrust of the change via the email notification 
alone. 

• Difficult integration across 3 programs 

o Getting to an integrated Cross-Program objective story was difficult to 
achieve as some entities were already ahead in their work and any 
vehicle changes resulted in significant SLS cost impacts; essentially 
put the onus on one Program to comply with the vehicle-focused 
[Vehicle Stability System] VSS design. 

Q24: Were there cost or schedule impacts due to communication glitches during 

the CR review? 
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Figure 5.12 Summation of SLS CR Survey Question 24 Results 


84 



From the previous question, an average of 78% of total survey respondents 
reported no communication issues, while 22% reported experiencing 
communication gaps and identified them. An interesting data point here was that 
previously, the CR53 respondents reported no communication gaps during the 
review; however, in this question, one out of the eight CR53 respondents felt 
there were resource impacts to cost or schedule and related them to 
communication gaps. Therefore, it was difficult to conclude that any cost or 
schedule impacts relate directly back to the 22% that recognized communication 
issues during the CR processing. However, strictly assessing the data as 
presented here, 80-88% of respondents, across all 3 CRs, who recognized 
communication issues, did not see those impacting substantial resources such as 
cost and schedule. The 12-20% of those respondents that did see cost or schedule 
impacts due to ineffective communication provided specific examples as reflected 
in the next question. 

Q25: If yes [there were cost or schedule impacts], what were they? 

• Time / Schedule 

o The CR [70] had to be delayed multiple times due to its size and 
complexity. 

o The amount of time needed to work through the [CR 70] Rev D 
impacts took valuable time that would have otherwise been used on 
other tasks. This created stress on the employees attempting to meet 
dates that were being pushed hard to keep the program, or Level II 


85 



schedule on track, without concern for the schedule impact to the 
element. 

• Cost 

o Unsure cost impacts were fully vetted by the design solution at the 
time of the PCB. (CR 53) 

o Getting to an integrated Cross-Program objective story was difficult to 
achieve as some entities were already out ahead in their work and any 
vehicle changes resulted in significant SLS cost impacts. (CR 82) 

• Cost and Schedule 

o Lack of understanding life cycle cost impacts of missed design 
influence decisions resulting in schedule delays. (CR 70) 

5.4.2 CR Problem Identification Analysis 

All the answers submitted by the survey respondents with respect to the 
problem identification questions were detailed in 5.4.1. An analysis of the 
effectiveness of the CR change control process by analyzing whether blatant 
and/or underlying problems were evident will be discussed in this section. 
Specifically, an assessment of how the desired answers selectable for the problem 
identification questions perfonned against the survey inputs from the respondents’ 
answers across all three CRs will be evaluated. What are desired choices when 
attempting to identify and rectify problems arising during the decision making 
process of the three CRs in study? Those were identified as participant 
motivation to review and ensure compliance, correct expertise identified for 
assessment, adequate time for review, adequate information accessible for review, 
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adequate bidirectional participatory discussion of technical inputs and risk 
mitigation and/or management, and finally, understanding the resulting decision. 
Table 5.2 lists these desired answers selectable and the resulting percentages for 
each CR. 


Table 5.2 Desired Selectable Answers for Problem Identification 


Desired Selectable Answers 

CR53 

CR70 

CR82 

Motivated for technical accuracy and compliance 
among Cross Program 

100% 

80% 

60% 

Needed no additional expertise 

100% 

83% 

80% 

Adequate time for review 

100% 

84% 

80% 

Adequate assessment from review 

67% 

97% 

100% 

Understood decision 

100% 

92% 

80% 

No communication gaps 

100% 

74% 

60% 

Observed no cost/schedule impacts 

88% 

87% 

80% 


The CR results for the desired problem identification questions ranged 
from a lowest value of 60% to a highest value of 100%. Evaluation of the three 
CR results with a 75% delineator revealed that the CRs were collectively above 
average for having adequate expertise and review periods assigned for CR 
evaluation, for identifying and/or mitigating impacts to budget or schedule, and 
for possessing a team consensus in the understanding of the resulting decisions. 
While most of the CR results favored the desired answers, there were, however, a 
few problems identified by each CR during the review, communication, and 
decision making process. The CR53 results showed a 67% rating for the 
assessment of comments during the Table Top Review process, CR70 narrowly 
missed the delineator mark with a 74% rating on communication, and the CR82 
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results were 60% both in motivation and communication. Specifics of these 
process hindrances were: 

• Adequate Assessment 

o CR53 respondents understood the resulting decision for the FTS 
architecture and considered the software upgrade workable, but one 
respondent felt future implications with the upgrade were not been 
adequately assessed. 

• Communication Gaps: 

o CR53 identified no communication issues while twelve CR70 
respondents and two CR82 respondents identified communication 
ambiguities with cost, schedule, review iterations, task team roles and 
responsibilities, and the integration process across the three Cross 
Programs. 

• Motivation: 

o All of the CR53 respondents identified technical considerations and 
coordination strategies with the Cross Program entities to establish a 
viable and safe FTS architecture as the motivation for the CR 
processing. Twenty-eight out of thirty-five respondents on CR70 
revealed similar motivation strategies. For CR82, three respondents 
shared a desire for technical accuracy and compliance across the Cross 
Programs whereas two respondents commented their motivation was 
“part of the job” which may or may not have been a positive 
motivator. 
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Legitimate problems were identified during the processing of the three 
CRs under study. Despite these problematic issues, the respondents reviewed, 
debated, and rationalized the decisional information as necessary to reach 
consensual decisions and programmatic approvals of the CRs. Overall, the 
decision making and communication within the CR processing was above 
average; however, based on the respondents’ feedback, there exist areas of 
improvement within the SLS CR change control process to be addressed and 
refined. 


5.5 Success Results and Analysis 

There were nine success-oriented questions pertaining to the three CRs. 
The results will be discussed in the following sections. 

5.5.1 Results per Question 

Q5: Do you believe you provided a needed contribution to the CR assessment? 
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Figure 5.13 Summation of SLS CR Survey Question 5 Results* 


Greater than 75% of the respondents across the three CRs felt they provided a 
necessary and beneficial contribution to the CR reviews. 
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Q10: If your answer to Question 5 was 'Yes', how do you feel your comments 
were received and dis-positioned? 


65% 60% 


81 


40% 


35% 


40% 

ill 


0% 0% 


0 % 



Well. Lots of discussion to 
make my comments 
understood. 

Fair. Minimal 
communication (mainly 
via email). 

Poor. My comments were 
ignored or dismissed. 

0 CR 53 

60% 

40% 

0% 

0 CR 70 

65% 

35% 

0% 

63 CR 82 

60% 

40% 

0% 


Figure 5.14 Summation of SLS CR Survey Question 10 Results 


All of the respondents felt their comments were adequately received and assessed. 
Q15: Do you think your comments were assessed adequately in the Table Top or 
Task Team Review process? 


97% 


67% 


33% 


3% 


0% 


100% 


■ 



No 

Yes 

eaCR 53 

33% 

67% 

□ CR 70 

3% 

97% 

SCR 82 

0% 

100% 


Figure 5.15 Summation of SLS CR Survey Question 15 Results* 


Between 67% and 100% of the respondents across the three CRs felt their 
comments were adequately assessed during the Table Top or Task Team Review 
meetings. The one exception in CR53 was explained in the following question. 
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Q16: If no, please explain. 


• The future implications [of FTS option 10A] were not given much 
consideration. The future configurations [were] not viewed as design 
considerations. 


o Solution was workable but did not adequately consider future 
implications or risk in operational scenarios. 

Q18: Did you fully understand the CR decision and its implementation plan? If 


not, what would have aided your understanding? 


100% 


92% 


0% 


8% 


20 % 



80% 


■ 



No 

Yes 

B CR 53 

0% 

100% 

□ CR 70 

8% 

92% 

SCR 82 

20% 

80% 


Figure 5.16 Summation of SLS CR Survey Question 18 Results 


Between 80% and 100% of respondents across the CRs understood the resulting 
decisions and corresponding implementation plans. Issues that impacted the 
understanding of CR70 and CR82 decisions were explained in the next question. 
Q19: If not, what would have aided your understanding? 

• A better big picture understanding of intent of CR [70] and what problems it 
was solving. 

• We were plowing new ground with what it meant to execute a hard T-0 design 
— that design is still in work after all these months. (CR 82) 
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Q20: To what degree did you agree with the decision? 


0% 0% 0% 


13% 


25% 


40% 
28% ^ 


50% 


59% 60% 


,13% 
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0% 


13% 


■ 



I 



Not at all (0%) 
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rationale (30%) 
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agree (80%) 
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agree (100%) 

0 CR 53 
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0% 

0% 
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0% 

0% 

0% 
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Figure 5.17 Summation of SLS CR Survey Question 20 Results* 


The majority of the survey respondents found the resulting decisions to be 
agreeable and workable. The 13% dissenting opinion for CR53 came from a 
respondent who disagreed with the FTS architecture decision but understood the 
rationale for the decision. The respondent commented: 

• While the option selected had some benefits, the other option was less 
complex, [had] no interfaces to deal with, and reduced the mass on the Core 
Stage. 

Q26: From your perspective, were your concerns with this CR dealt with 
effectively? 
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Figure 5.18 Summation of SLS CR Survey Question 26 Results* 


Among the three CRs under study, the majority of the respondents commented 
that they felt their inputs to the CR change process and their contributions to the 
resulting decisions were adequately and effectively handled. However, there were 
a few dissensions with respect to CR70. One respondent answered this question 
with “No, not at all” and added a comment that the respondent did not review the 
CR. Therefore for assessment purposes, no review of the CR equated to no 
concerns with the processing of the CR. For the next category of “Somewhat but 
not adequately”, a couple of CR70 respondents provided the following comments 
to explain why they felt this answer was appropriate with respect to their inputs to 
the CR: 

• One of my comments was rejected due to the elimination of [Data 
Requirements Description/Definition] DRD without my knowledge. 

• Cost swept under the rug, as they have been for other changes. Level II 
requirements drove cost that the elements had to find a way to make happen 
with no additional money. 
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Q27: From your perspective, how would you describe the decision process with 
respect to this CR? 


56% 52% 60% 


0% 3% 


20 % 


5% 


22% 23% 20% 


22% 17% 



Very difficult 
and/or 

frustrating and 
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Figure 5.19 Summation of SLS CR Survey Question 27 Results* 


An averaged majority of 91% across all three CRs observed the decision process 
to be effective. This percentage broke down to: 22% of all respondents viewed 
the decision process as effective but inefficient, 56% viewed the decision process 
as sufficiently effective and somewhat inefficient, and 13% of the respondents 
viewed the decision process as highly effective and very efficient. For the other 
end of the spectrum, an average of 9% of the survey respondents found the 
process to be difficult, specifically expressing the decision process to be very 
difficult, frustrating, and ineffective. The following data addressed the difficult 
and ineffective process examples identified by the survey respondents. 

• CR Review Process 

o The CR and Table Top process is long and time consuming. There are 
areas that can be eliminated and improved. (CR 82) 


94 



o The perseverance of the professionals that we had working the product 
was the only reason the CR [70] made it through the process at all. In 
other words, the process did not help, the work got done (mostly) in 
spite of the process. 

o Never given enough time to review any CRs. (CR 70) 

• CR Scope / Content 

o The most inefficient part, in my opinion, was caused by people adding 
documents to their scope without updating the DRL. This happened 
before the CR [70] was sent out for review. Granted, it is a by-product 
of the phase of program we were in, and things have tightened down 
since that time. 

o Inefficient because there were so many changes associated with this 
CR [70] it was difficult to keep up with understanding changes 
provided by other commenters. 

• CR Cost Impacts 

o This CR [70] had to be withdrawn and re-released (as SLS-00070R1) 
due to cost impacts from various entities. The process worked, but it 
might have been more efficient had cost impacts been addressed 
during the task team process. 

o Hard to present an objective story because of out of synch schedules. 
Any design changes were major cost impacts to the Program. (CR82) 
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5.5.2 CR Success Analysis 

All the answers submitted by the survey respondents with respect to the 
success-oriented questions were detailed in 5.5.1. An analysis of the success of 
the overall decision making and communication practices exercised within the CR 
change control process will be discussed in this section. As conducted on the 
previous survey assessments for the process and problem identification questions, 
an evaluation of how the desired answers selectable for the success-oriented 
questions performed against the actual survey answers from the respondents 
across all three CRs will be analyzed. The desired choices for the success- 
oriented questions were identified as those where the respondents felt they made a 
necessary and valuable contribution to the review and resulting decision, where 
their comments and contributions were acknowledged and adequately assessed, 
where they understood and agreed with the resulting decision, and lastly, where 
they exhibited a high degree of confidence in the effectiveness of the decision 
making process. These desired answers selectable and the resulting percentages 
for each CR for that answer are listed in Table 5.3. 


Table 5.3 Desired Selectable Answers for Success 


Desired Selectable Answers 

CR53 

CR70 

CR82 

Provided needed contribution to CR assessment 

75% 

89% 

80% 

Adequate comment disposition 

100% 

100% 

100% 

Adequate assessment from review 

67% 

97% 

100% 

Understood decision 

100% 

92% 

80% 

Moderately agreed to completely agreed with decision 

63% 

87% 

100% 

CR concerns acceptably to highly effectively assessed 

100% 

89% 

100% 

Sufficiently to highly effective decision making process 

78% 

69% 

60% 


96 





The CR results for the desired success-oriented questions ranged from a 
lowest value of 60% to a highest value of 100%. Evaluation of the three CR 
results with a 75% delineator revealed that the CRs were collectively above 
average for having respondents who felt they provided a needed contribution to 
the CR assessments, felt their comments were assessed highly effectively, and 
understood the resulting decisions. A majority of the CR results favored a desired 
success (i.e., >75%) indicating positive influences of success for the decision 
making process; however, there were four less desirable occurrences in comment 
assessment, decision agreement, and decision making process effectiveness. 
Respondents for CR70 and CR82 showed higher percentages of satisfaction in 
how their comments were assessed during team reviews and in agreement with 
the resulting decision than the CR53 respondents where the CR53 results were 
67% and 63%, respectively. In the case of decision agreement, the CR53 
respondent moderately agreed with the decision but completely understood the 
rationale and thereby supported the resulting decision. Despite the CR53 less 
desirable results for comment assessment and decision agreement, the CR53 
respondents ranked the overall decision making process as sufficiently to highly 
effective and yielded the highest ranking amongst the three CRs. This data result 
seemed illogical since CR70 and CR82 had exceeded the 75% delineator in all the 
desired areas except this one decision making process effectiveness descriptor that 
yielded percentages of 69% and 60%, respectively. While the SLS CR change 
control process worked and decisions were made, there were, however, observed 
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scope, ambiguity, and resource issues that hindered the overall decision making 
and communication process as reflected in these recorded comments: 

• If disciplines would have done a due diligence and understood the 
affordability tenant upfront this [DRL] would have been a better product. 

• This [CR70] was a large, difficult CR to deal with because of its scope. But, 
given that scope, I feel the actual CR process itself worked fairly well. Cost 
impacts were generated which were dis-positioned at the boards. Ultimately 
the CR was re-scoped due to a board decision to not absorb the original costs. 
It would have been better to work changes in scope (such as new documents, 
etc.) with decision-makers earlier, instead of waiting until this CR, but 
considering that this did not happen the CR process itself did what it was 
supposed to do. 

• Any changes [for CR82] were major cost impacts to the Program. 

Despite the observed hindrances, the level of respondent knowledge, skill, 
and involvement applied to the technical evaluations, discussions, and decisions 
proved to be sufficiently effective as reflected in the survey results. 

5.6 Summation of Main Findings 

The data results across all three CRs under study revealed that all 
comments from the CR respondents were adequately acknowledged and assessed. 
The data results showed that an average greater than 80% of the CR respondents 
were not only actively involved in the CR deliberations concerning the technical 
changes including risks and consequences of those risks, but they also felt they 
made necessary contributions to the resulting decisions. Additionally, an average 
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greater than 90% of the CR respondents said that they not only fully understood 
both the CR decisions and the implementation plans, but they also agreed with the 
decisions. While the majority of the respondents felt involved in the reviews and 
formulation of the consensual decisions, there were less than 20% of the 
respondents who voiced not being involved in impact discussions or aware of the 
official approval of the CRs. 

When the CR respondents were asked to describe the decision process, an 
average of 96% of the respondents across all three CRs described the decision 
making process as effective. Nine percent, however, observed the process as 
difficult and ineffective stating inefficiencies in the time allocations and the SMEs 
assignments to the review of CRs. With respect to communication within the 
decision making process, an average of 78% of the CR respondents did not 
experience or observe any issues in communication. Fifteen percent of the 22% 
who confirmed experiencing communication issues said those issues resulted in 
cost impacts due in large part to the inability of entities to perfonn cost 
assessments correctly or consistently for the technical changes under review. 
Communication proved effective in the formulation of decision recommendations 
to be provided to and approved by the final decision authority; however, the 
communication of the final decision and approved CR back down the hierarchy 
was less effective. 

Assessment of the CR data results identified four areas of process 
improvement. These include the official notification of CR review and CR final 
approval, the definition of the CR scope and/or purpose, and the inclusion of all 
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reviewers in all facets of the CR process. While 67% of the respondents said they 
were on the official email from the CM Release Desk, if it had not been for 
informal emails and verbal requests from other parties, the CRs would not have 
received adequate reviews. Similarly, the notification of the CR final approval 
was inefficient because 16% of the respondents said they never received official 
notification the CRs were approved. Unclear scopes and/or expectations for the 
technical review within the CRs were recurring comments from the respondents. 
And lastly, less than 20% of the respondents, who had no comments or non-trivial 
comments such as editorials, were not involved in review meetings where impacts 
were discussed. 

5.7 Key Decision Drivers 

The respondents ranked attributes such as design/performance, cost, 
schedule and risk in priority (top, second, third, lowest or not considered). 
Weights were then applied to achieve an overall score. 


Attribute Rank Weight 

Top Priority 4 

2 nd Priority 3 

3 rd Priority 2 

Lowest Priority 1 

Not Considered 0 


The overall score per weighted attribute per CR was then normalized to 
find the percentage breakdown for the attributes per CR. From Figure 5.20, it 
appears cost followed closely by design/performance were the decision drivers for 
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the FTS architecture change in CR53. Design followed closely by cost and 
schedule were the decision drivers for CR70’s baseline update to the list of all 
data requirements required for the SLS Program. Finally, design and cost tied for 
the decision drivers for the change to the Core Stage forward skirt umbilical in 
CR82. Across all three CRs under study, the resulting key drivers were 
design/performance and cost for making the final decisions. 




Design / 
Performance 

Cost 

Schedule 

Risk 

0 CR 53 

30% 

32% 

17% 

20% 

□ CR 70 

29% 

27% 

26% 

18% 

SCR 82 

29% 

29% 

24% 

17% 


Figure 5.20 Key Driver Results for SLS CR Decisions 
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CHAPTER VI 


CONCLUSIONS AND RECOMMENDATIONS 

6.1 Conclusions 

Evidence from the study of three decisions within the SLS CR change 
control process suggested that SLS Program has consciously reviewed and 
applied lessons learned from previous large-scale programs to the execution of the 
SLS Program. Specific evidence findings of and recommendations for the SLS 
decision making and communication processes will be discussed in this chapter. 

6.1.1 Constellation Lessons Learned 

Evidence suggested the SLS Program applied the lessons learned from the 
Constellation Program to its decision making and communication processes. The 
ESD within NASA Headquarters was accountable for ensuring that integration 
was executed effectively, efficiently, and affordably across the three Cross 
Programs. Specifically for the SLS Program, the program’s Configuration 
Management and Data Management (CM/DM) repository provided insight into 
the program-level documentation that defined how the SLS Program would 
implement Cross Program and internal technical integration from design, 
development, design analysis, test, and certification (SLS Integration Plan, 2013) 
to systems engineering and integration roles, responsibilities, and processes 
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specific to the integrated launch vehicle design and implementation (SLS Systems 
Engineering Management Plan, 2013). 

The SLS decisions under study were strategic with deliberations among a 
group to reach a consensual recommendation presented forward to SLS Program 
Management for final approval. The communication appeared to have been better 
executed within the SLS Program than on Constellation; however, the SLS 
communication process still needs improvement within the overall decision 
making process. 

6.1.2 SLS Decision Making Effectiveness 

The SLS Program aimed to balance timely decision making at the 
appropriate levels within the CR change control process. Before discussing the 
SLS decision making effectiveness, evidence of how the SLS process aligned 
with the plethora of research from decision theory experts such as Hickson, Dean, 
Sharfman, Papadakis, Thompson, Reagan, and Shrivastava will be discussed. 
6.1.2. 1 SLS Decision Making Dimensions 

The three dimensions of strategic decision making (i.e., procedural 
rationality, politics, and complexity) were evident in the CR decision study. 
Kelley & Thibaut (1969) theorized that the quality of information available to 
decision makers is one of the most important detenninants of successful decision 
making. Evidence of procedural rationality relies upon analysis of this 
information relevant to the decision in making a choice (Dean & Sharfman, 
1993b). Survey results showed that all the respondents had readily available 
material for review; however, the review time was an issue. The deficiency of 
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adequate time to review is considered by Janis (1989) to be a negative influence 
that can lead to decision analysis issues. Fortunately, this cognitive constraint 
was not dominate, per the survey results, and did not appear to have hindered the 
decision making process. Survey results also showed that all the respondents 
contributed necessary CR comments with no comments being ignored but rather 
all effectively assessed. Much review and discussion went into the comment dis- 
positions and the analyses of impacts, risks, and constraints to arrive at the viable 
strategic decision recommendations presented to the PM for final approval of 
each CR. 

Each of the Cross Programs (SLS, Orion, GSDO) across three NASA 
Centers (MSFC, JSC, KSC) comprised its own specific processes, its own groups 
of experts with varying motivations for involvement in the CR review, and its 
own idiosyncrasies factoring into decision making. Consequently, politics was 
introduced. An example of political imbalance of influence from the survey 
results was “getting to an integrated Cross-Program objective story was difficult 
to achieve as some entities were already out ahead in their work and any vehicle 
changes resulted in significant SLS cost impacts; essentially put the onus on one 
Program to comply with the vehicle-focused [Vehicle Stability System] VSS 
design.” An example of political contention over objectives was “cost impacts 
were not fully vetted by this design solution at the time of the PCB.” Another 
example, “the method of doing changes to the DRL should be given mandated 
undivided attention. . .suggestions of how this should work... the document owner 
should be knowledgeable enough to make the decisions on changes to be 
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made. . .then the document should be updated. . . .provided. . .with changes 
highlighted. . .and with a meeting to go through the document in a scheduled order 
so commenters can attend when areas they are interested in are discussed. The 
Table Top method of going through the spreadsheet one comment at a time was a 
waste.” 

Evidence of complexity included a confusing or unclear scope, intricate or 
ambiguous data, and diverse or conflicting views, interests, and/or opinions 
among the decision makers. All of these complexity issues were evident in the 
survey results. For instance, one survey respondent cited, “a better big 
picture. . .of intent of DRL CR [70] and what problems it was solving” would have 
aided understanding. Another cited, “[CR 82] was an urgent CR whose intent 
was not well communicated. . .couldn’t figure out the thrust of the change via the 
email notification alone.” There were also examples of intricate or ambiguous 
data cited. For instance, “we were plowing new ground with what it meant to 
execute a hard T-0 design. . .with that design. . .still in work after all these 
months.” Other respondents cited examples from “several significant cost 
impacts had to be worked out through the board process” to “CR [70] had to be 
delayed multiple times due to its size and complexity. . .it was released. . .and then 
re-released. . .as R1 . . .there was some question as to what was retained in the 
comments from R0 review to Rl” to “the amount of time needed to work through 
the [CR 70] Rev D impacts took valuable time that would have otherwise been 
used on other tasks.” Fastly, survey examples of complexity stemming from 
diverse or conflicting views and/or opinions were “unclear cost impacts from 
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some parties in the days leading up to the boards. . .resulted in "piling on" at the 
board meetings” to “the SLS Task Team review is... confusing to most outside 
organizations... [understanding] who participates, how are they chosen, how are 
they notified.” 

Analysis of the evidence showed all three dimensions of strategic decision 
making present in the assessments of the three SLS CRs. Politics and complexity 
were evident in the multi-disciplined grouping of personnel with diverse skills, 
mental models, and motivations evaluating each CR’s intricate and sometimes 
unclear data. Application of decision tools fostered the rational processing to 
reach consensual decisions. 

6.1.2.2 SLS Decision Making Model 

Evidence suggested that the SLS Program decision making process 
tracked to a hybrid managerial autocracy and adaptive planning decision making 
model. From the managerial autocracy model perspective, a large amount of 
power and authority rested with the single key manager (i.e., SLS PM) who made 
all SLS strategic decisions himself with technical assistance from several 
subordinates (Shrivastava, 1983); however, the SLS PM did not bias the style and 
preferences for reaching the decision or stifle the use of system tools, procedures, 
and/or experience of the SLS organization in providing the necessary technical 
assistance. From the adaptive planning model perspective, plans were viewed as 
guidelines that were modified depending on the current analysis of issues. 

Specific to the CRs, each one differed in technicality. One CR was software 
related, another hardware related, and another pertained to documentation. 
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Qualified SLS discipline experts systematically evaluated the technical merits of 
each proposed change in an effort to achieve efficient and adaptable solutions to 
the problem each CR presented (Shrivastava, 1983). Evidence suggested that the 
SLS CR change control processing “involved systematic participation by relevant 
members who could handle the technical complexity, evaluating risks and 
environmental constraints, and effectively communicate information in which to 
achieve viable strategic decisions and implementation plans that yielded solutions 
to problems” (Shrivastava, 1983). Each CR reached final approval by the SLS 
PM implying the SLS PM agreed with the decision recommendations provided by 
the technical disciplines. Based on this analysis, the SLS Program decision 
making process appeared to pattern a hybrid managerial autocracy and adaptive 
planning decision making model. 

6.1.2.3 SLS Decision Making Strategy 

Evidence suggested that the decisions under study were traceable to a 
judgment decision strategy employing the tractable-fluid decision making mode. 
The survey results showed that the right disciplines were identified and included 
at the appropriate levels/phases of the process. One respondent cited, “additional 
expertise is always needed, but that need has to be balanced with the cost of 
managing excessive input. . .the balance achieved for this revision was 
reasonable.” Another cited, “while the review period for this CR was sufficient 
for one CR of this magnitude, this CR was not the only one under review... that is 
the price of a tight schedule. . .multiple changes being reviewed simultaneously by 
the same experts. . .do the best we can. . .request extensions when. . .cannot 
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accomplish an appropriate review. . .there's never enough time to do it all.” 

Results also showed that ah of the comments received were acknowledged with 
the vast majority of the comments adequately dis-positioned. The survey results 
reflected a couple of exceptions: “One of my comments was rejected due to the 
elimination of DRD [Data Requirements Description] without my knowledge.” 
and “Cost swept under the rug, just as. ..for other changes. . .Level II requirements 
drove cost that the Elements had to find a way to make happen with no additional 
money.” Similar results were evident for the process effectiveness in that the 
majority of the survey respondents thought the overall process was effective with 
a couple of exceptions. Respondents perceiving the process as difficult, very 
frustrating, and ineffective cited, “[It is] hard to present an objective story because 
of out of synch schedules. . .design changes were major cost impacts.” and “. . .the 
magnitude of change and the importance of the document created an environment 
where there was frustration at many levels. . .perseverance of the 
professionals. . .working the product was the only reason the CR [70] made it 
through the process.” The vast majority understood and agreed with the resulting 
decisions. One respondent who did not agree with the resulting decision cited, 
“While the option selected had some benefits. . .the other option was less complex, 
no interfaces to deal with, and reduced the mass.” This respondent did not agree 
with the decision but understood the rationale in which to support the decision 
made. Evidence further revealed a medium-to-high level of complexity and 
politics across the three CRs. One respondent cited communication issues 
stemming from “cost impacts not fully vetted within the CR review.” Complexity 
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of the large, complicated CR [70] made communication a struggle resulting in 
cost, schedule and workload impacts. A corresponding survey comment cited, “A 
large, difficult CR to deal with because of its scope. . .people are overloaded, 
schedule is tough, decisions are hard, and everyone's not going to be happy in the 
end. . .but. . .given that scope. . .the actual CR process itself worked fairly well.” 
Also political comments reflecting an imbalance of influence and contention over 
objectives made the politics within the decision making process high. 
Consequently, analysis of the evidence suggested the common decision making 
strategy for the three CRs approximates to the tractable-fluid decision mode 
where the CR reviewers collaborated and scrutinized the details to reach 
understanding and negotiated the resulting decisions (i.e., a judgment strategy). 
6.1.2.4 SLS Decision Making Effectiveness 

Evidence suggested that the decision making within the CR change control 
process worked well from five perspectives: participatory identification, review 
notification, reviewer involvement, input assessment, and decision fonnulation. 
Survey evidence revealed the appropriate competencies were indeed identified as 
necessary participants in the CR reviews, and those individuals felt they were 
adequately notified of the reviews and in turn provided essential [or valuable] 
inputs to the CR reviews, impact discussions, and the decision recommendations 
that were presented to the final decision authority. Survey results further revealed 
that all received comments were effectively assessed, that all of the participating 
individuals understood the resulting CR decisions with the majority of them 
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(>80%) agreeing with the decisions, and lastly that they considered the overall 
decision making process to be effective and efficient. 

While each CR completed its approval process differently, evidence 
suggested that the SLS decision making process was less process dependent than 
typical systems engineers might expect. The variation in CR approval process 
appeared to have not impacted the overall decision making process and success of 
each CR; instead, the tailored approach, as opposed to a standard process rigor, 
was appropriate for each CR. As long as the process matched the needs of the 
decision makers and an effort was made to get all needed individuals involved, 
different processes appear to be used effectively. 

Evidence from the CR study suggested the SLS Program was sufficiently 
effective at making strategic decisions via a comprehensive decision making 
process. As a guideline, a decision making process matters; however, a process 
that is adaptable to a project’s needs (i.e., size, complexity, risk posture) is ideal. 
Of all the survey responses, only one commenter disagreed with this premise 
citing, “the process did not help, the work got done in spite of the process.” The 
NASA Systems Engineering Handbook (2007) specifies an evaluation be made to 
determine the magnitude of the change required, and then the process be tailored 
to address the issues appropriately. This approach promoted effectiveness to the 
process as opposed to brainlessly following a process just to follow a process. 
However, with any leniencies provided in a process, attention to thorough review, 
communication, and execution of the negotiated, tailored process must always be 
consciously exercised. 
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Evidence showed a variation in fonnality within the SLS decision making 
process. The initial contact with all the stakeholders was generally formal with a 
10% to 17% amount of informality. More infonnality was evident in the 
notification and communication at the Table Top / Task Team review level where 
informality ranged between 17% and 32%. It seemed more appropriate for this 
review level to be less formal. One observation was that the communication of 
notification at this review level appeared to be less inclusive (depending on the 
comments). For instance, if reviewers had no comments or had minor comments 
such as editorials, then the reviewers were not always part of the discussions on 
comment dispositions. This raised a flag indicating a potential communication 
issue since approximately 25% were not involved in discussions of any impacts 
stemming from the dis-positioned comments from the CR reviews. Some 
respondents with no comments would not be included in the discussion, therefore, 
missing a decision and/or impact discussions leading to the formulation of a 
decision. 

The overall process differed for each CR. One difference was in the array 
of individuals with varying levels of knowledge, skills, interests, and workload 
tasks assigned as reviewers. Secondly, the CR subject matter, data products, 
control boards (i.e., CECB, PCB, JICB), and review periods varied amongst the 
CRs due to project time constraints and other dependencies. Lastly, the process 
differed with the institution of the new Task Team review concept for one CR 
under study. In each case, the same decision making process was referenced for 
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technical guidance but tailored to the specific CR need and decision maker 
expertise. 

Evidence showed that the same resources were expended for the three CRs 
under study. A respondent shared, “Each organization tries to place those with 
the most expertise on the CRs.” Additionally, most of the survey comments 
suggested that lack of resources such as time and budget were the reason 
additional expertise would have helped in the CR reviews. One respondent 
commented that “there were always too many CRs in the system with too little 
time to complete as one would like”. Other comments such as: “SMEs 
are. . .overburdened with CR reviews”, “multiple changes. . .reviewed 
simultaneously by the same experts”, and “never enough time to do it all” served 
as a result of the overwhelming workload of many CRs in general, not due to a 
specific CR. 

Evidence showed cost and design/performance strongly influenced the 
decisions. Those involved in the CR processing felt their contribution was value 
added to the decision making and approval of the technical change. The survey 
results revealed no more than 25% had no need to contribute. It appeared the SLS 
Program would rather commit a type II error (i.e., asking a few more people to 
comment who do not have a comment to make) than a type I error (i.e., that is 
failing to ask someone who might have an appropriate comment). 

All comments received during the CR reviews were deemed acceptably 
dis-positioned. Most CR participants completely or moderately agreed with the 
resulting decisions. In the case of CR53 there was one person who, while 


112 



understanding the rationale for the decisions, thought there was a better 
alternative that was less complex, cheaper in the long run to operate, and was 
afraid the decision made was a short term and not the best long tenn decision. No 
one disagreed completely. 

The Reagan and Rohrbaugh (1990) study was referenced for the 
evaluation of the decision making process and the resultant decisions from the 
process to detennine whether the SLS decision making process was goal- 
centered, adaptable, participatory, and data-based. The SLS decision making 
process was adaptable and participatory; however, the process was not 
consistently goal-centered or data-based across the CRs under study. When 
reviewers did not know or understand the scope or intent of CR in review, then 
the process was not adequately goal-centered. Similarly, when the magnitude of 
the change was too much to process within a defined time constraint, then the 
process was not adequately data-based. Were the resulting decisions efficient, 
legitimate, supportable, and accountable? Evidence showed that no one 
completely disagreed with the resulting decisions; therefore, the decisions were 
considered efficient, legitimate, supportable, and accountable since procedural 
rationality was applied to logically produce a necessary effect for each CR 
change. 

6,1.3 SLS Communication Effectiveness 

Hackman (1990) theorized that three “enabling conditions” (i.e., sufficient 
group effort, adequate knowledge and skills possessed by group members, and 
appropriate performance in decision making strategies) exert positive influence on 
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group performance through the mediation of communication and interactions. 
Eikenberry (2005) theorized that individuals communicate the right information at 
the right time in the right way to make an effective decision. Habermas (1998) 
theorized that communication between people must be fulfilled to coordinate 
actions effectively for the purpose of satisfying needs and employing an effective 
decision making process. The question was, did the SLS disciplines meet these 
communication guidelines for effective decision making? Evidence suggested the 
CR reviews and resulting decisions appear to be well-vetted, understood by all, 
and agreed to by the majority of the CR reviewers. The decisions seemed to also 
be well-communicated to the Program Manager for approval; however, the 
dissemination of the CR approvals were not so well-communicated to the 
stakeholders and/or organizations. It was a disturbing result that 16% never 
officially received notification of the approved CRs. The decision making process 
needs improvement for proper dissemination of decisions. 

Table Top and Task Team reviews were the mode of communication for 
the CR processing. These reviews were face to face meetings with audio (i.e., 
teleconference) interactions. Per McGrath (1984), this hybrid mode is less 
restrictive on communication and provides increased opportunity to exchange and 
utilize information in arriving at solutions / decisions. Jarboe (1996) theorized 
that group involvement in decision making increases the amount of information 
available to the group, increases commitment to the decision, improves 
dissemination of that decision, and increases the quality thought throughout the 
process. While evidence suggested the decision making process to be effective, 
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the communication was not a total positive influence in the process. Evidence 
revealed 16% of the survey respondents never officially learned of the approved 
CR decisions, indicating a communication gap within the process. 

Communication within the decision formulation and determination appeared 
adequate with the dissemination of the resulting decision lacking. This result 
aligned with Eikenberry (2005) who theorized the widest gap in decision making 
to be in communicating decisions. 

The Constellation Program comprised a 10-Center team. Communication 
was a documented challenge for that program. The SLS Program spans a smaller 
grouping of integration efforts across 3 -Centers; however, with respect to the CR 
processing and decision making, communication exhibited challenges to the 
Program. Evidence suggested that the decision making process was inefficient in 
resource (i.e., time and workload) allocations and in communicating decisions. 

6.2 Recommendations 

The overall assessment of decision-making and communication as 
evidenced by analysis of these 3 CRs was positive. The SLS Program has 
improved and incorporated lessons learned from recent past programs. There are, 
however, a few recommendations to be made which would further strengthen a 
successful decision-making and communication process. 

There is a need to include all involved parties in the discussion of the 
comments. The mere fact that an individual did not have a comment was reason to 
exclude the individual from the comment discussion. On the surface this seems 
appropriate; however, there were respondents who mentioned that changes were 
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made to a CR during discussion of a comment which were not known to those not 
included in the discussion. Involving all parties requires additional resources up 
front but may solve issues in the long term. 

The effectiveness of the decision process was hindered by the difference 
between NASA’s schedule and external entities’ schedules. A decision making 
approach that has NASA schedules more in synch its counterparts’ schedules 
would facilitate needed changes and quick responses. 

There were multiple comments about the workload within and across CRs. 
The time and resources to review, understand, and completely assess all the CRs 
was extremely limited. It was clear the SMEs felt the pressure to respond quickly 
and thoroughly but acknowledged that this had the risk of overlooking a problem 
or implementing a conservative answer and/or comment. Faster is not always 
better. 

Approximately 20% of the respondents never officially learned of the 
approved CR decisions. Inefficient communication resulted in cost and schedule 
impacts. Based on the communication issues, an establishment of a culmination 
meeting at the end of the CR decision process to close the communication loop 
would be beneficial. 

There were concerns about individuals’ understanding, skill, and timing of 
life cycle cost assessments. There is a need to train Cross Program personnel on 
how and when to perfonn cost and schedule impact assessments within a review. 
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6.3 Thesis Research Contribution 


The culmination of this thesis research will contribute to the body of 
knowledge by providing a better understanding of the decision making process 
within the Systems Engineering discipline. Fonnal Systems Engineering 
processes are documented, but the informal implementation of Systems 
Engineering are not fully understood. While this thesis focused on the formal and 
informal interactions and practices employed by the NASA Marshall Center to 
investigate, collaborate, and negotiate viable strategic decisions within the SLS 
Program, the knowledge and implementation of decision making and 
communications captured within the thesis can be effectively applied to Systems 
Engineering practices within any type of organization (i.e., government, 
cooperate, academia, etc.). 
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APPENDIX A 


SURVEY ADMINISTRATION 


Subject: Decision Analysis Survey 

On behalf of Garry Lyles, your completion of the Decision Analysis Survey below is 
greatly appreciated. 

The University of Alabama in Huntsville is following 3 different Change Requests to 
understand SLS decision making processes. One way to gather information to aide this 
effort is by studying how each of these 3 CRs was introduced, discussed, approved and 
then communicated. We would like to survey those involved in each of these CRs to 
better understand the discussion patterns, the approval process and the resulting decision, 
and communication of that decision. This is not a critique of the decision making 
process, but a study to determine key drivers in decision making. We are trying to 
identify aspects that strongly influenced the decisions made and those aspects which are 
more flexible. The survey should take between 4 and 7 minutes to complete. 

It has been placed on Survey Gizmo to protect anonymity. No names will be used in the 
reporting of the data or conclusions. Please take a few minutes to complete the survey at 
the link below for CR00070 or forward to your delegate for this CR as appropriate. 

For Questions please contact: Karen Hicks at kch0039@uah.edu or Dawn Utley at 
utleyd@uah.edu . 

The survey can be found at the following link: 

http://www.surveygizmo.com/s3/1349312/SYSTEMS-ENGINEERING-PRQCESSES- 

SURVEY-SLS-DECISION-PROCESS-FQR-CR-SLS-00070 


Some of you may receive more than one survey based in your participation in the review 
of the CRs selected for the survey. If so, please fill out a separate survey for each CR in 
which you were involved. 
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APPENDIX B 


SURVEY INSTRUMENT 


Change Request (CR) SLS-OOXX Review 

1 

Who (person or group) generated the CR? 


2 

How were you notified of this CR for 
assessment? (Check one.) 

□ Direct 
email to 
review 

□ Email from 
DLE / EDLE to 

review 

□ Verbal from 
DLE / EDLE to 
review 

□ Other 

3 

If ‘Other’ checked above, please elaborate. 


4 

What was your motivation in reviewing this 
CR? 


5 

Do you believe you provided a needed 
contribution to the CR assessment? (Check 

one.) 

□ No. I did not need to 
contribute to this CR and had 
no comments. 

□ Yes 

6 

Would the assessment have benefited from 
additional expertise or input? (Check one.) 

□ No 

□ Yes 

7 

If ‘Yes’ above, what expertise would have 
improved the CR assessment? 


8 

Do you feel you had adequate time and/or 
readily available CR related materials to 
perform an assessment of this CR? (Check 
one.) 

□ No 

□ Yes 

9 

If ‘No’ above, what hindered your review? 


10 

If your answer to Question 5 was ‘No’, then 
skip to Ouestion 1 1 ; otherwise, how do vou 
feel your comments were received and 
dispositioned? (Check one.) 

□ Well. 

Lots of discussion to 
make my comments 
understood 

□ Fair. 
Minimal 
communication 
(mainly via email) 

□ Poor. 

My comments were 
ignored or dismissed 

11 

Were you involved in the discussion of any 
impacts stemming from the dispositioned 
comments from the review of the CR? 
(Check one.) 

□ No, not at all 

□ Yes, informally 
(i.e., through DLE, 
EDLE, LSE, CE, 
etc.) 

□ Yes, formally via 
direct Table Top / 
Task Team / Control 
Board Review(s) 

12 

If ‘No’, please explain. Then skip to 
Question 17. 


13 

If ‘Yes’ in Question 11, how were you 
notified of the Tabletop or Task Team 
Review for this CR? (Check one.) 

□ Direct 
email to 
review 

□ Email from 
DLE / EDLE to 
review 

□ Verbal from 
DLE / EDLE to 
review 

□ Other 

14 

If ‘Other’ checked above, please provide 
info. 


15 

Do you think your comments were assessed 
adequately in the Tabletop or Task Team 
Review process? (Check one.) 

□ No 

□ Yes 

16 

If ‘No’ above, please explain. 


17 

When did the decision / approval of the CR 
officially get to you (approximate mm, yy) 
or NEVER? 
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18 

Did you fully understand the CR decision 
and its implementation plan? (Check one.) 

□ No 

□ Yes 

19 

If ‘No’ above, what would have aided your 
understanding? 


20 

To what degree did you agree with the decision? (Check one and provide any comments.) 

□ Not at all (0%) 

□ Disagreed 
but understood 
rationale (30%) 

□ Ambivalent 
and workable 
(50%) 

□ Moderately 
agree (80%) 

□ Completely 
agree (100%) 

Comments? 

21 

In your opinion how were the following attributes used in making the final decision of this CR? Rank the attributes, 
(where 0=Not Considered, l=Top Priority, 2=Second Priority, etc in a pull-down menu). 

|Cliek to select a rank| 
Design/P erforman 
ce 





|Click to select a| 
Cost 

|Click to select a| 
Schedule 

|Click to select a| 
Risk 

|CIick to select a rank 
Other: 

Communication 

22 

Were there any gaps in communication during the CR 
review? (Check one.) 

□ No 

□ Yes, minor 

□ Yes, major 

23 

If ‘Yes', what were they? 


24 

Were there cost or schedule impacts due to 
communication glitches during the CR review? (Check 
one.) 

□ No 

□ Yes, minor 

□ Yes, major 

25 

If ‘Yes’, what were they? 


Overall Assessment of the CR Decision-Making Process 

26 

From your perspective, were your concerns with this CR dealt with effectively? (Check one and provide any comments.) 

□ No, not at all 

□ Somewhat 
but not 
adequately 

□ Acceptable 

□ Moderately 
effective 

□ Highly 
effective 

Comments? 

27 

From your perspective, how would you describe the decision process with respect to this CR? (Check one and provide any 
comments.) 

□ Very difficult 
and/or frustrating 
and not effective 

□ Difficult and 
somewhat 
effective 

□ Effective but 
inefficient 

□ Sufficiently 
effective and 
somewhat 
inefficient 

□ Highly 
effective and 
very efficient 

Comments? 

28 

If you have anything you would like to share about this 
CR review, please do so here. 
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